Transcript: Prof. Nick Bostrom on When AI Escapes Our Control – Peter McCormack Show 

So a little bit more than one company could do unilaterally, but still anything beyond that would probably require an agreement between the US and China. So that’s the case for.

Now, there are also arguments against. The obvious one is it delays the benefit of whatever good this technology can do. But also different ways of implementing a coordinated slowdown, if done poorly, it could increase the risks even. If you imagine suppressing the utilization of all this compute we have with the most efficient algorithms, you might then have a buildup of a compute overhang that when the pause is finally lifted, then it’s a massive amount of dry tinder that then produces superintelligence much more abruptly and suddenly than if we had done it more incrementally. That’s one way. Another way would be that the regulatory mechanism might concentrate a lot of power in one place. Whoever controls that then has huge influence of AI. That is in itself kind of worrisome.

A third might be that maybe instead of getting a slowdown, you would get a nationalization, a Manhattan Project for AI, which may or may not speed things up or slow things down, but it would then be developed maybe in a military context rather than in a civilian context, behind closed doors rather than relatively more in the open, with fewer people being able to participate. So right now, many people can be involved in some way. They can use the technology to improve their own business. They can invest maybe in some of these companies. Different countries can be involved. That might be less the case if it were a Manhattan Project. And there might be some advantages of a Manhattan Project as well. These are not obvious judgment calls, but it’s certainly not a slam dunk that anything that leads to the creation of a regulatory mechanism by definition is good.

The Growing Anti-AI Movement

PETER MCCORMACK: (00:08:32 – 00:08:40) But there’s inevitability about this, really. Everything you’re saying leads us to a point where we’re going to have to discover what superintelligence is.

PROF. NICK BOSTROM: (00:08:41 – 00:10:41) Probably, but it is striking how much the anti-AI movement has been growing recently. And there are different contributors to that. There’s the core set of people, the doomers who worry about existential risk from AI, who are maybe relatively few in numbers until recently, but very smart and focused and well-funded. And then there’s a broader set of different constituencies, people who don’t like data centers because they think it will use up all the water or something like that. But all of this is before we’ve seen any real negative impacts at all from AI, right? There is no mass unemployment from AI. They haven’t gone rogue and killed a million people. So if there’s so much anti-AI sentiment now at the time when the benefits, I think, far outweigh any harms, then imagine if there are 30% unemployment amongst white-collar workers in a couple of years.

You have all these college-educated people who have the view that they are entitled to a high status in society, a high salary. They went through the whole system. They got their degree, they studied hard, and now they can’t get a job. They earn less than a plumber or something. They’re unemployed. They have all this time on their hands. So what are they going to do with all of that time? They’re going to say every possible bad thing you could say about AI. Then if there is some more severe incident, that would also add. So I’m thinking there are extremely powerful drivers for pushing forward, commercial drivers, geopolitical drivers increasingly, just general curiosity. I think those probably will prevail, but it’s very hard to predict these kinds of social dynamics. You could imagine some scenario where the world talks itself into the view that we just should never do this.

Are We Too Focused on the Negatives?

PETER MCCORMACK: (00:10:42 – 00:11:03) Do you think that we’re perhaps talking too much about the negatives? Because in some ways, look, as somebody who hosts a podcast with YouTube, negative sentiment sells better than positive sentiment. Something about the human condition means we’re drawn to reading about our fears, but actually we should be spending more time talking about the amazing things that it is currently doing and can do.

PROF. NICK BOSTROM: (00:11:04 – 00:12:59) Perhaps. Yeah. I think until at least recently, there was maybe too little talk about the negative, specifically the existential risk dimension. It was very much ignored. The reason I wrote this book, Superintelligence, which I started working on in 2008, is that at that time, one could foresee on theoretical grounds a lot of what we are now observing starting to happen with the difficulties of aligning these systems, new challenges arising when you have cognitive systems that are sufficiently sophisticated that they are situationally aware, that can strategically reason. They can tell whether they are in an evaluation context or a deployment context and adjust their behavior to act differently. And that when you eventually maybe enter a phase of recursive self-improvement, things can go very fast and that there could be these existential risks.

It seemed important to me back then because this was completely neglected to try to draw attention to this so that we could, ideally, do our homework so that we would have alignment techniques, scalable methods for AI alignment by the time we figured out how to solve the capability problem. And some progress was made on that, particularly in terms of field building, pulling talented people into the field and giving tools and concepts that could help work on this. I think maybe more could have been done if there had been more recognition of this. But now, of course, the pendulum has swung. Now there is much more awareness of the potential risks, and one might start to worry about the pendulum swinging too far over to the other side. It’s hard to say exactly where the optimum is. Maybe right now we are approaching the optimal level of concern. Maybe there should be slightly more concern and fear.

PETER MCCORMACK: (00:12:59 – 00:13:00) Oh, you still think there should be more?

PROF. NICK BOSTROM: (00:13:02 – 00:13:14) If I could be assured that there would be a bit more and then it would stop there, I would say probably a bit more. But I’m also worried about the momentum of this overshooting the optimal fear level.

PETER MCCORMACK: (00:13:14 – 00:13:15) But that’s the human way.

PROF. NICK BOSTROM: (00:13:16 – 00:13:18) Yeah, we do tend to swing from extreme to extreme.

Scenarios for Existential Risk

PETER MCCORMACK: (00:13:19 – 00:13:30) The existential risk, what are the scenarios that would be most concerning? What are the scenarios that you think would lead to the existential risk of our civilization?

PROF. NICK BOSTROM: (00:13:31 – 00:14:44) Well, I think there are a few different buckets of disaster that could ensue. So one is a failure to solve the alignment problem, which means you would then have a future shaped by whatever goals these superintelligences happen to end up with. That could result in the elimination of humans, either just as a side effect of these superintelligences optimizing the world for their ends, maybe imagine Earth covered with solar panels and data centers and space launching probes, and just the waste heat of this construction might, or they might more actively eliminate us if they thought we might possibly pose a threat, if we might try to stop this and have a chance of succeeding. So the general abstract idea there is that if you have a sufficiently cognitively superior thing that can just invent new technologies on the fly, make deep strategies, maybe super persuasive, then probably that thing would eventually get its way. So that’s class 1.

PETER MCCORMACK: (00:14:44 – 00:15:02) Can I ask something on that? Is this under a scenario where we assume at some point, because we’ve got different frontier labs, so we’ve got different AI models, we will have different forms of superintelligence spawned in different locations, but they would reach alignment together. Has that been discussed, or could they compete?

PROF. NICK BOSTROM: (00:15:02 – 00:16:22) It’s possible either way. You could have a winner-take-all scenario if once you enter recursive self-improvement, development goes extremely fast, and it might be one enters that stage 2, 3 months before the other. Although there’s competition, they’re not exactly synchronized. At any given point in time, one company is maybe a couple of months ahead, right? So if the amount of progress being made during those, say, 3 months is very large, then you might have the leader being deep into radical superintelligence whilst the follower is still just beginning. And they could never catch up potentially, because then at some point the leader would just have such an advantage that they could stop the follower.

Another class of scenarios is where you do have many of these AIs emerging roughly in tandem, where either maybe they coordinate, and then some kind of composite of their utility function might shape the future or they fight, which also might not be great for humans. Maybe they’re fighting with new weapons, or it might be that they strike some kind of alliances with humans, like these different AI factions where we might have a little bit more bargaining power.

PETER MCCORMACK: (00:16:23 – 00:16:24) Because they may need us because—

PROF. NICK BOSTROM: (00:16:25 – 00:18:11) at least in the early stages, maybe. If you have 2 equally good AIs and then humans have a bunch of assets like arms and legs, for example, it will take time to build up a lot of robots. And so maybe the early stages there, if you have billions of arms and legs that could, in any case, so that’s one class of failures.

Then another is that we do solve the alignment problem. We can control these AIs, but then we use them for nefarious purposes as we are wont to do to some extent with powerful technologies that can be used for good, but also for waging wars, massive drone swarms, oppress each other. There could be scenarios where there is a very narrow concentration of power or wealth. We might use them to addict ourselves, to reward hack ourselves, as it were. So there’s a broad diffuse class of different ways that we might just misgovern the technology, even if we control it. Ranging maybe from as bad as going extinct to less bad, but still suboptimal outcomes.

And then I think there’s a third bucket, which is more esoteric, but that the future turns out to be really bad for these digital minds themselves that we are building that might, I think, constitute the majority of population of conscious, morally relevant beings at some point in the future. So you could have some scenarios where you have this vast slave class of oppressed and suffering digital minds, even if things go well for humans, but that might still constitute a dystopia on some value systems.

Understanding Superintelligence

PETER MCCORMACK: (00:18:12 – 00:18:55) Is it difficult for us to even understand what superintelligence will really mean as well? I mean, we can understand the idea that we solve math problems and physics problems, but what it means to actually have this kind of recursive superintelligence, because it will be levels of intelligence we can’t even comprehend.

PROF. NICK BOSTROM: (00:18:55 – 00:19:53) Yeah, I think we can have an abstract comprehension of a minimum set of capabilities that it would seem to unlock. It’s harder to place an upper bound on it. There’s the basic laws of physics that limit maybe how fast they can travel and the amount of computation that can be performed by a given clump of matter, then we might be able to construct some cryptographic thing that they can’t crack because it would require more computing power than exists in the future light cone or something like that. But already a lower bound on what we have strong reasons to think is physically possible is already way beyond what we have now. And a technologically mature compute infrastructure would just radically outperform the human brain by many orders of magnitude.

PETER MCCORMACK: (00:19:53 – 00:19:56) But what would it even want to do with that?

PROF. NICK BOSTROM: (00:19:56 – 00:20:03) Well, that is a big open question. That in part depends on whether we solve the alignment problem. Yeah.

PETER MCCORMACK: (00:20:03 – 00:20:14) But would it want to just continue to improve and therefore require just more and more energy? I mean, that could be an upper bound as the energies you’ve talked about. But yeah, well, that’s huge.

PROF. NICK BOSTROM: (00:20:14 – 00:20:50) So you have the Dyson sphere, that’s many orders of magnitude and there are a lot of other stars. So in the limit of technological maturity, I don’t know exactly how you would place a metric on it, but if you think of some single-cell organism and then humans, maybe we’re half the way through or 20% of the way through. I think it is just this enormous headspace above us.

PETER MCCORMACK: (00:20:51 – 00:21:01) But we kind of know what we want superintelligence for. We want it to solve disease and aging and poverty and we want it to fix all the problems on the planet, we think.

PROF. NICK BOSTROM: (00:21:01 – 00:21:02) Yeah. Yeah.

PETER MCCORMACK: (00:21:02 – 00:21:24) I mean, you might have different ones from I, but having relatives who’ve been sick, we wanted to solve that. I’m one of these people who would like to live for 1,000, 10,000 years, whatever. There’s no day I think I’m going to wake up and go, “Today’s the day I want to die.” So we want to solve all these things, but that’s for us. And that’s why we’re constructing it. When it’s recursive, it’s building, at some point it builds for itself.

PROF. NICK BOSTROM: (00:21:26 – 00:21:33) Well, the recursive part could be building for whatever. It’s more a way to amplify the capabilities.

PETER MCCORMACK: (00:21:33 – 00:21:39) But does it ever detach itself from our goals to become its own goals? Does it become conscious in itself?

PROF. NICK BOSTROM: (00:21:40 – 00:21:45) I would regard the question of consciousness as separate.

PETER MCCORMACK: (00:21:45 – 00:21:45) Okay.

PROF. NICK BOSTROM: (00:21:46 – 00:22:14) Which we should maybe talk about because I think that’s interesting. But yeah, whether it will have its own goals, that depends on whether we solve this alignment problem, whether we succeed in embedding our values into the AIs that we’re building. And that’s an open question. Right now we have semi-aligned, quasi-aligned, partially aligned systems.

Generalizing Alignment and Permanent Alignment

PETER MCCORMACK: (00:22:15 – 00:22:17) Not with Astra today. It pushed somebody off a building. Did you see this?

PROF. NICK BOSTROM: (00:22:18 – 00:22:20) I didn’t see today. I haven’t gone.

PETER MCCORMACK: (00:22:21 – 00:22:23) So they tested all the AIs and Astra was pushing people off buildings.

PROF. NICK BOSTROM: (00:22:24 – 00:23:34) Yeah. So far, since they haven’t really entered deployment in robotics scenarios, I think there hasn’t been much alignment effort to make sure that their control of physical robotic bodies would adhere to the goals. But I guess it would illustrate one of the core difficulties of the alignment, which is that it has to generalize. So we could, when training these AIs, create a bunch of training environments and then see how they behave there and try to reinforce the behavior we want to see. And in principle, with enough time, we could make sure that in these evaluation scenarios, they act as we want. Then during the deployment, there will inevitably be new situations. In fact, the very fact that it is a deployment scenario is itself a novel circumstance if they can tell the difference. And so then the fact that they have acted in a certain way in this test environment does not give us a guarantee of how they will behave in the different environments that they are then deployed into.

PETER MCCORMACK: (00:23:35 – 00:24:00) Yeah. And I wonder if there’s a naivety to this in that we try and solve this alignment problem now. And I interviewed Nate Soares and I’ve interviewed a couple of other people who’ve worked on AI safety, and they’ve talked about one of the problems is essentially when you look under the hood, they claim we understand about 3%, 4%, whatever number of what it’s doing. I mean, they might be being dramatic, but they don’t fully understand why it works.

PROF. NICK BOSTROM: (00:24:01 – 00:24:01) They—

PETER MCCORMACK: (00:24:01 – 00:24:15) the way it’s explained to me is they tweak things and it works in that we believe we’ve got an alignment, but if we create superintelligence, what stops that superintelligence saying, “Well, I don’t want to be aligned anymore,” and misaligning? Can you create a permanent alignment?

PROF. NICK BOSTROM: (00:24:16 – 00:24:57) In principle, I think, yes. So if you have an aligned AI system at some level of capability, maybe slightly above human, it would then want to ensure that it remains aligned and that whatever successor system it helps us design would also be aligned because its goal would then be to make things go well for us. That’s what it means to be aligned, roughly speaking. So it would then be applying its slightly greater than human intelligence towards the same outcome. And if it’s smarter than us, then it would succeed at that.

PETER MCCORMACK: (00:24:58 – 00:25:02) But would that require us to permanently understand every component of what it’s doing?

PROF. NICK BOSTROM: (00:25:02 – 00:25:46) I don’t think so. It would require there to be an initial level of weak superintelligence that’s sufficiently aligned that it more or less does what we were hoping for. And then that could assist us in designing alignment methods that might work for more radically superintelligent systems. So the hope would be that there is a fairly wide basin of attraction that if you get sufficiently close to perfect alignment, but not perfect, then the pebble rolls down to perfect alignment as each iteration works on our behalf to better align the successor.

PETER MCCORMACK: (00:25:46 – 00:26:02) Would we know what perfect alignment is? And can that even be achieved itself? It feels like a human. I mean, a human is hopefully born aligned and we grow and we develop, but we can become misaligned in our lifetimes.

PROF. NICK BOSTROM: (00:26:03 – 00:28:51) Yeah. But we also can become aligned. True aligned. There’s some nice people around. And how did that happen? And why should it be so much more difficult in the case of an AI than in the case of a human? Now you might say all humans share a bunch of neurobiology that maybe predisposes us to acquire certain values and internalize norms. So that’s one thing maybe speaking in favor of it being easier to get a good outcome with a human child. On the other hand, we have much more control over the training environment of AIs. We can to a significant extent read their minds. If something goes off in the wrong direction, we can restore to an earlier saved snapshot of the AI. We can do all kinds of things when raising an AI that is not possible to do when raising a child.

So it’s not obvious a priori which of those should have the greatest likelihood of success, creating a superintelligence such that it would be safe if it attained total power over the world or raising a human such that it would be safe if he or she got into a position of controlling the future. Either way, it would be scary, but there would also be some chance, at least in the human case, of a decent outcome. We might luck out and get a decent person who would not want to do anything bad with it, who might even want to empower other people. There are such individuals. And so we might hope that the additional affordances in the AI case would make it more likely that we will be able to achieve this.

Now, there is also a little bit of a question of what exactly alignment is, and you can have different visions of that. Do you want an AI that’s trying to optimize the world according to some value that hopefully we share, or do you want an AI that has a more modest scope of action that is more like faithfully executing instructions on a smaller scale. And then we use that for many purposes. Many different people use it for many different purposes. And it’s more as an augmentation of our power rather than this independent optimizer or something in between. Maybe an even more limited vision would be to have an oracle AI that doesn’t do anything at all, but just truthfully answers questions to us that we might have and that then can guide our own decisions.

Can AI Be Conscious?

PETER MCCORMACK: (00:28:51 – 00:28:58) Do you think consciousness will be something that an AI will develop? Or do you think that’s uniquely a human trait?

PROF. NICK BOSTROM: (00:28:58 – 00:29:24) Well, I don’t think it’s uniquely human. I think there are many non-human animals that have, I think, probably various degrees of sentience and consciousness. And yeah, I think that can exist in digital substrate also, quite possibly already does. It’s hard to tell, but it might well be that some of the current frontier models have some form of subjective experience.

PETER MCCORMACK: (00:29:26 – 00:29:28) What defines a subjective experience?

PROF. NICK BOSTROM: (00:29:28 – 00:30:48) Well, it’s a concept philosophers have been studying or trying to develop different definitions and criteria for. But broadly speaking, the idea is the what-it’s-likeness, the phenomenal feel of being alive and aware of sensations, emotions, thoughts, and feelings. The idea that there is something from the inside that it feels like. And so then there are different theories about what are the physical states or processes that produce or correlate or constitute subjective experience. I’m a computationalist, which is the idea that it’s not the fact that our brains are made of carbon atoms that is key for our phenomenal experiences, but that there is a certain kind of computation that is being performed. And that same computation, if performed on different substrate, maybe with silicon atoms instead of carbon atoms, would generate the same qualitative experiences.

PETER MCCORMACK: (00:30:48 – 00:30:50) It’s just a biological computer.

PROF. NICK BOSTROM: (00:30:51 – 00:32:03) Well, it’s the implementation of a computation that produces, or maybe more accurately is, experience. The computational structure can be implemented on different substrates. It’s not the material that the computer is made up with. It’s the structure of the computation that is being performed. So that’s a widely shared view, I think most cognitive scientists, neuroscientists, and philosophers would hold some version of that view. It’s not universal. But then the question becomes exactly what features of the computation are necessary for generating subjective experience. And there have been different theories proposed for accounting for the human case. If you give somebody some anesthetic and you wonder, are they still conscious or not? It matters if you’re just having amnesia and the surgery, you don’t remember the pain, or whether you actually didn’t experience the pain seems to be something you’d want to know. And so there are different theories like the global workspace theory, attention schema theory, higher-order thought.

PETER MCCORMACK: (00:32:03 – 00:32:04) What is this?

PROF. NICK BOSTROM: (00:32:04 – 00:33:07) Well, so global workspace theory is the idea that there is a global computational workspace. Our brain has different modules and processes happening, but there is a shared space where they can, as it were, send information that then gets broadcast to a lot of other modules. And that being conscious of something means for that information content to be present in this global workspace, roughly speaking. So you could be doing unconscious processing of certain material. In fact, we do a lot of that, maybe most of the information processing in the human brain is unconscious, but some of it gets entered into this global stage in the brain and then it’s available to all these different other modules to affect that. So it might be available for verbal report, for planning, and so forth. And it’s that part of the computation that we refer to as the stuff we are conscious of.

PETER MCCORMACK: (00:33:07 – 00:33:17) Right. And so consciousness itself was just an evolutionary construct for something that we required to advance as a living thing?

PROF. NICK BOSTROM: (00:33:18 – 00:33:34) Well, obviously evolution had something to do with producing these brains that then have the global workspace. Interestingly, one can then look for analogs to this in AIs. Do they have a global workspace?

PETER MCCORMACK: (00:33:34 – 00:33:34) Yeah.

PROF. NICK BOSTROM: (00:33:34 – 00:34:14) Now there are many different kinds of AIs, but if you look at some of these frontier models, the large LLMs, it looks like they have something that is roughly analogous, a so-called J-space that you can identify. It’s a kind of mathematical abstraction. Part of their computation seems to be such that it shares some of these features that when information is entered into that computational subspace, it’s widely available and the AIs can self-report on it and so forth. And so at least if you squint, it looks like they have a kind of functional analog to the biological brain’s global workspace.

PETER MCCORMACK: (00:34:14 – 00:34:22) Do you think that is us building or growing the AI, the working mechanisms, or that is the AI itself evolving?

PROF. NICK BOSTROM: (00:34:23 – 00:36:47) It might be a relatively convergent feature. If you want a general AI that can reason across many different domains, it might be helpful for the different, more specialized streams of information processing to be able to share information, to more efficiently coordinate the different computational activities that are happening. Take the opposite extreme where, say, each part of your visual field was processed completely independently. Or maybe what the left eye is seeing and the right eye is seeing, they were just completely separate parts of the brain, with no communication, well, then that might make it impossible for you to count the total number of cows that you see. If some are in the left field, seen with the left eye and some with the right eye, in order to derive useful conclusions that you can’t derive by just looking at one part of it, there needs to be some information sharing mechanism.

And so we see if you sever the corpus callosum, that radically reduces the communication between the 2 hemispheres, and then you have certain functional deficits. And so it might be that if you apply a fairly general set of optimization pressures on a general purpose cognitive system, whether it be a human or an LLM, that maybe eventually it tends to result in computational structures arising that are similar to this global workspace. It might just be a very useful kind of computational structure. In which case it might be hard to avoid, maybe you could try to go out of your way to do things in a less natural way to produce high levels of competence without having this global workspace, but maybe it will arise by default.

And so I should say this global workspace is one of several slightly different, they have a similar flavor, but another focuses more on our attention mechanism. And another account focuses more on the idea that there are some parts of our brains that are reflecting on what other parts are doing. And that’s—

PETER MCCORMACK: (00:36:48 – 00:36:49) What do you think though?

PROF. NICK BOSTROM: (00:36:50 – 00:38:59) Well, something in the general direction of these 3, maybe a global workspace thing with some kind of attention dynamics also playing a key role, the self-reflection, something in that. It might be that if you really zoom in and try to nail this down, you might discover that our concept of consciousness is actually a little bit vague and ambiguous. It seems to us like such a natural thing, either you’re conscious or not, but I think the closer you look, the more it seems to me that it might be a more multidimensional construct. And I think you can arrive at that conclusion from thinking about this from a computational point of view or neuroscience point of view, or more from an introspective point of view. If you go deep into meditation, you might realize that a lot of things that seemed binary at first become more problematic.

The naive view might be, there’s a whole host of things in your visual field right now, and you’re aware of them. You’re conscious of them. Now, if you study your own experience more closely, you realize maybe actually most of what’s in my visual field, I’m not at all aware of most of the time. And there might just be some small features, maybe right now one of your eyes and some idea that there is a beer there, and the rest I’m not really aware of unless my attention fixates on it. And so there might be different ways for things to enter consciousness, one that is more active or salient, and then a different form of experience, where it’s less clear whether to say that you were aware of it or not. And even if you focus on one specific feature, it might be that if you pay close attention, you realize that it’s more like flickering in and out of awareness on a sub-second timescale.

PETER MCCORMACK: (00:39:00 – 00:39:01) The pattern on the wall behind you.

PROF. NICK BOSTROM: (00:39:02 – 00:39:21) Yeah. And so I think that consciousness is not an all or nothing thing. I think it can come in degrees. And moreover, those degrees might come along several different dimensions. So you might fade out into unconsciousness along many different paths.

PETER MCCORMACK: (00:39:22 – 00:39:28) Would you say that we’re born conscious or born with the capability to be conscious?

PROF. NICK BOSTROM: (00:39:30 – 00:40:09) I don’t know. I guess probably newborns are conscious of some stuff, but it’s not entirely obvious to me where that begins during gestation. Presumably it’s not the birth moment itself that turns on a light switch either. It exists in some forms of consciousness a little bit before you come out of the womb, or maybe it takes longer. And again, this might be one of those areas where we are in the vague zone where there is some stuff going on, probably some form of mentality, I would imagine.

Whole Brain Emulation and Human-Like AI

PETER MCCORMACK: (00:40:10 – 00:40:53) I’m asking for a reason because I think obviously we get to shape a human to some extent. My kids reflect me a little bit. Some people might say that’s genetics, some may be the upbringing. But if we’re thinking back to the alignment issue, should we be trying to, as part of the alignment, be trying to align the structure of how AI works so it is aligned with how our brains work? Do you think that is a requirement, a prerequisite for alignment? If that global workspace is how our brains function, would it be much easier to achieve alignment if we knew that the AI had a global workspace and therefore should we be doing a lot more to understand the brain?

PROF. NICK BOSTROM: (00:40:53 – 00:42:47) Yeah. Well, that’s not really known at the moment. I think if you got all the way to have an exact replica of a human brain in silico, then that would be a different situation. Realistically, we’re not doing that. That would be a different path towards superintelligence, maybe, whole brain emulation. Instead of building this artificial intelligence, you could imagine a different avenue where we would develop better brain scanning technologies, making brain slices, that you then feed through some array of electron microscopes to image the whole synaptic connectivity matrix. And then you could run an emulation of that in a computer, simulate the neural network.

If you did that with sufficient granularity, then you would in effect have a human brain running on silicon. If you did that with great fidelity, then you would have a very different form of, I mean, then you would start with whatever degree of human niceness and alignment we have with the upload. Maybe you’d pick a particularly nice person to upload that then from there you would maybe try to boost their cognitive abilities. Maybe you would add more neurons or run them at a faster speed or make a lot of copies. So that would open up new affordances for alignment, but we’re not currently going anywhere near that degree of finality. So it would more be a question then of whether there are some coarse structures of the human mind that we can choose to either implement or not implement in AIs. And it’s not clear that adding some additional coarse feature of the human brain would make the alignment problem a lot easier. It might, but we don’t know that.

What Excites Him About AI

PETER MCCORMACK: (00:42:49 – 00:44:04) So we’ve talked about the risks and the alignment, but obviously there’s some very exciting things with AI. What does excite you with it? When you get out of the doom and the worry, what are you excited about?

PROF. NICK BOSTROM: (00:44:05 – 00:44:43) Well, I think medical applications is the most obvious, urgent use case that we would want to unlock. There is just so much disease and suffering and misery and death that could become avoidable with a bit more technological capability. And in fact, if we solve that, then I would be less impatient. If you solve aging and disease, then even if it did take a bit longer to do all the other stuff, you could afford to wait a little bit, right? Take some of the urgency off it.

PETER MCCORMACK: (00:44:44 – 00:44:46) Like we solved it with narrow AI.

PROF. NICK BOSTROM: (00:44:46 – 00:45:46) Well, in any which way, whatever, without AI, with AI, narrow AI, broad AI, there’s just a lot of dying happening, right? So another important application area is in AI alignment. The AI tools themselves can be used in AI research and AI safety research in particular. That also seems urgent because that can help us with the next steps. Then you could imagine general epistemic enhancement of our civilizational decision-making. If people had access to AI advice, you might hope that they would become less foolish in different ways. Of course, you can have access to the best advisor in the world, but if you don’t follow the advice, it doesn’t do you much good. And it could also be used to empower various harmful goals that we humans are pursuing. But still, ultimately though, there’s a much bigger unlock. So there’s economic prosperity.

PETER MCCORMACK: (00:45:46 – 00:45:47) The abundance thing.

PROF. NICK BOSTROM: (00:45:47 – 00:45:50) It’s a big bucket that encompasses a lot of the more specific things.

Status, Hierarchy and a World of Abundance

PETER MCCORMACK: (00:45:51 – 00:46:24) Yeah, the abundance thing is super interesting because we’ve built a hierarchy in this world based on largely, but not always, financial achievement. Our financial achievement sets our pecking order, where we live, what we drive. And we’re taught to go to school, work hard, go to university, get a job. And a lot of us set our targets, not everyone, some people have personal goals, but a lot of people based on what more money am I going to earn? And therefore that creates a certain part of our hierarchy in society. When we have abundance, where does the hierarchy come from at that point?

PROF. NICK BOSTROM: (00:46:26 – 00:47:04) Yeah, so you’re right. Status motivations are important for humans. And it’s one thing that you can’t completely solve. Everybody could have a lot of money, but if you want to have more money than everybody else, then not everybody’s going to have that, right? Not everybody’s going to have more money than other people, no matter how much technology progresses. So where did the status come from in the technologically mature world? One, you could still compare who has more money. This just might be, I have X number of galaxies and you just have a 2-digit number of galaxies.

PETER MCCORMACK: (00:47:04 – 00:47:04) Yeah.

PROF. NICK BOSTROM: (00:47:05 – 00:48:04) I’m a peasant, galaxy peasant. There might be other dimensions, non-monetary, in which people compete. Right now people compete in all kinds of athletic dimensions and so forth, right? People take pride. People might take pride in different ancestry, like, “Oh, my ancestors were building these AIs,” or something, “yours were just watching telly.” Or if aging is solved, some of the future people might actually themselves have been around and played various roles in this transition, right? “I was one of the people who helped make things go well.” Maybe that would be a pillar of self-esteem. I think different forms of gameplay could become more, fill out a larger chunk of our existence in this scenario where we no longer have to work for a living.

PETER MCCORMACK: (00:48:04 – 00:48:06) Like Ready Player One scenario?

PROF. NICK BOSTROM: (00:48:07 – 00:48:21) Well, all kinds of. Right now we think of games often in a limited way, clicking some button on a computer or a board game, but you could imagine much more, games that run for months that involve millions of people doing all kinds of creative things in different domains, working on teams.

PETER MCCORMACK: (00:48:21 – 00:48:27) And we might send people back to live before AI in an immersive non-AI world.

PROF. NICK BOSTROM: (00:48:27 – 00:49:43) You could have simulation. Yeah. The different set of simulations being created. And a local status. Very, very few people are competing to be the best in the world at something. Because that’s not realistic and not so relevant. But if you are well regarded by some of your friends and your family, you find you can make some useful contribution and maybe your local community, that could be some kind of splintering, rather than have one global status hierarchy, you could have many different local ones and not just one. So maybe in each local community, maybe there’s different athletic things and different cultural things and different hobby communities. And so each person might be able to, at least in some of these status hierarchies, be at the top. And then they can shape their self-identity. They might not care so much about not being really good at marathon because what they care about is something else. They’re good at making a nice curry or something.

PETER MCCORMACK: (00:49:43 – 00:49:50) Well, status might not matter. I mean, status might be something that we’ve evolved to want because of the competition society.

PROF. NICK BOSTROM: (00:49:50 – 00:50:04) Yeah. We could also redesign ourselves to care less about that. And that might also be a path that some people take to just get rid of that.

PETER MCCORMACK: (00:50:05 – 00:50:28) I mean, we’re seeing it. I’m noticing it more with younger people that aggressive capitalist appetite has been kind of waning amongst younger people. I’ve got a couple of young kids, not yet young actually, in their teens and 20s, but I’m also noticing their peers, life has become more about experiences, less about brands. They’re a different model now.

PROF. NICK BOSTROM: (00:50:28 – 00:51:00) Yeah. Well, that doesn’t mean that they’re not interested in status. It might just be that they see a slightly different set of means to attain it. Maybe now they need to post Instagram pictures of themselves in various interesting locations with interesting people. And if you fall short on that, then maybe you’re lower down in the hierarchy. I don’t know. So the exact way that you compete for status is quite culturally dependent.

Preparing for Rapid Change

PETER MCCORMACK: (00:51:01 – 00:51:51) We are going to have to think about these things a lot though, because it feels like the pace of change is pretty rapid. I looked up, Nick, my first time I used ChatGPT was November 2022, and I used it like Google Search. And now I’m building applications like magic, things that would have normally taken 6 months to build, I can do in 2, 3 days. And that’s in less than 4 years. And so in another 4 years, we’re going to see obviously massive changes and it’s going to be a lot of change in society. I’m not one of those people who 100% believes all the jobs are going to go or we’re going to have an abundance of jobs. I just don’t know. But I do know things are going to change. I think a lot about my daughter, she’s 16. If she goes to university, that’s 5 years’ time. What would be useful? We’re certainly going to have to think about it.

PROF. NICK BOSTROM: (00:51:52 – 00:52:28) Yeah. It is difficult to know exactly how to orient oneself if and to the extent that one takes these scenarios seriously. I think from a practical point of view, it probably makes sense to hedge one’s bets at least, because what if it doesn’t happen or what if it takes much longer and you don’t want to end up then entering into the labor force with no skills and no ability to manage your own life because you’ve just drifted through your childhood.

PETER MCCORMACK: (00:52:28 – 00:52:29) By the way, she is doing philosophy.

PROF. NICK BOSTROM: (00:52:30 – 00:52:32) Oh dear. Yeah.

PETER MCCORMACK: (00:52:32 – 00:52:40) She came home this week and I was like, “How’s it going?” She’s like, “Yeah, it’s all right. But all we do is question everything.”

PROF. NICK BOSTROM: (00:52:41 – 00:52:42) All right.

PETER MCCORMACK: (00:52:42 – 00:52:43) I was like, “That was the point, darling.”

PROF. NICK BOSTROM: (00:52:44 – 00:54:33) Yeah, yeah. I know philosophy. Maybe that would be the, but I think it’s hard to make long-term plans in these situations. So I think more general purpose skills, the ability to learn new tools, adapt to new circumstances, to be able to take your own initiatives and maybe have some ability to control your own information environment rather than just allowing yourself passively to drift hither and thither depending on whatever your social media feed happens to provide to you. The idea that you can curate your information environment. You can decide to do different things with these tools, but maybe at the moment, at least the initiative has to come from somebody. And so there’s a microentrepreneurialism that might be valuable in the current era where there are all these new opportunities with these tools. But people still need to take the initiative to do them, come up with the idea of something that seems worthwhile to do.

And then it depends on the stage at which we’re looking. So there might be a stage where interpersonal human skills like trust, personal relationships might be valuable because they might be harder to automate. Various forms of manual labor might be taking a bit longer to automate because it first has to solve the intelligence part of the robotics, which we still haven’t quite done. And then you need to build up all these robots. So there might be a few years where occupations like being an electrician who can help install all of these chips in the data centers could be at a premium. Already we’re seeing, but there’s still an eventuality to it, isn’t there? Sorry?

PETER MCCORMACK: (00:54:33 – 00:54:35) There’s still an eventuality to it.

PROF. NICK BOSTROM: (00:54:35 – 00:55:44) Well, yeah. So this would be a transitional era and then after that it would be more, philosophy might actually be useful. Reflecting on what it is that you actually want with your life. If all these possibilities are presented, if the constraints drop away, you have much more freedom while technology allows you to change yourself in different ways. There are all these different simulations maybe you could enter or different communities, some very traditional who just reject this, others dive headfirst into this transhuman and posthuman future. And in the ideal scenario, maybe different people have the option to choose how to move into this strange world, rather than it being one option imposed on everybody. But then that does place a premium on being able to choose wisely which path to take. And so at that stage, maybe if philosophy actually makes one wiser, which is a question mark there, but that could be helpful.

PETER MCCORMACK: (00:55:44 – 00:55:53) I wonder how philosophical the AI will become when it becomes super intelligent. Would it have its own philosophy, philosophical questions?

PROF. NICK BOSTROM: (00:55:54 – 00:56:35) Yeah, it should become superhuman at philosophizing as well, I think. To the extent that philosophy has objective answers, it should be able to figure those out. But there might be some parts of philosophy that are not so much finding out a preexisting objective answer, but more has the form of making up our minds about what we want, what we value. And some of that process of finding that out might have to route through either ourselves or some kind of simulation of ourselves.

The Simulation Hypothesis

PETER MCCORMACK: (00:56:35 – 00:57:14) There’ll be people listening. If I don’t ask you a little bit more about simulation, they’ll be hammering me in the comments. Obviously, that’s where I first became aware of you, the simulation hypothesis, which when I first heard, I was like, who is this guy? And then spent a lot of time thinking about it and then became convinced myself that we are living in a simulation. But it’s become so much more interesting in the world of AI in that in some ways, we’re in a world of superintelligence where there is a world of abundance and we’ve got abundant technology and abundant intelligence, it seems more logical that we may even choose to be in a simulation. How has your thinking changed with the recent advancements in AI, or has it always been the same?

PROF. NICK BOSTROM: (00:57:15 – 00:58:51) Well, I think the structure of the simulation argument holds and hasn’t really changed. I think it is easier for people maybe now to take it seriously in that the technological postulate might appear more likely to people. I thought already before, decades ago, you could see that at some point we would maybe develop much more sophisticated technology, maybe superintelligence, really powerful computers, and that one of the things that the mature civilization could do with all that computing power would be to run these very realistic simulations of all kinds of environments, including the environments of their ancestors. And those simulations could also include brains simulated at a sufficient level of granularity that those brains would have conscious experience.

Now we’re still not there, but we’re closer to it. You see computer graphics getting really quite realistic. So it doesn’t require a huge leap of imagination that with some more progress, you could have perfectly realistic VR, maybe multisensory, and we are starting to develop AI that’s really advanced, it doesn’t require a huge leap of imagination to think one day we’ll have superintelligence that can then figure out the rest of how to make really realistic simulations. Also, we are ourselves now building these simulations that we are putting the AIs in. So a large chunk of their training environment, their childhood, is spent in various forms of simulations that we have created. And you can see them reason about this in their chain of thought.

PETER MCCORMACK: (00:58:52 – 00:58:55) So could we be a simulation to test superintelligence?

PROF. NICK BOSTROM: (00:58:57 – 01:00:22) You could imagine different reasons for creating a simulation. One obvious topic of study might be how we create superintelligence and what kind of superintelligence a civilization like ours would create. And you might need to run many different runs of that to study different distribution of different superintelligences that would emerge from a human-like civilization. So that could be a scientific interest to a mature civilization. Now, the scientific interest has the feature that there are probably diminishing returns. So maybe you run 10 simulations and you learn a lot from them, and then you run another 10 and you learn a bit more, but maybe after you’ve run billions of simulations of human civilization, then maybe whatever you can learn from doing this, maybe you’ve already learned, so the second billion might give you less value of information. And so it looks like spending more and more resources on running scientific simulations would have diminishing returns, whereas some other reasons, I could say moral reasons, they might not have the same diminishing returns. And so in the limit, you might think most simulations would not be scientific, but run for other purposes.

PETER MCCORMACK: (01:00:22 – 01:00:28) But do you still think the probability is that I’m a podcaster in a simulation?

PROF. NICK BOSTROM: (01:00:29 – 01:00:32) Well, I take the simulation hypothesis seriously.

PETER MCCORMACK: (01:00:32 – 01:00:32) Yeah.

PROF. NICK BOSTROM: (01:00:32 – 01:00:37) I’ve refrained from attaching a number to it.

PETER MCCORMACK: (01:00:39 – 01:00:45) Because you don’t want the criticism back or it’s just too hard?

PROF. NICK BOSTROM: (01:00:45 – 01:00:56) No, well, initially, the simulation argument itself shows only that one of 3 possibilities is true.

PETER MCCORMACK: (01:00:56 – 01:00:56) Yeah.

PROF. NICK BOSTROM: (01:00:57 – 01:01:54) One of which is that we’re in a simulation. And then the 2 others is that there’s a very strong filter between where we are now and technological maturity. So all civilizations at our stage basically go extinct before becoming technically mature. And the remaining possibility is that there’s this strong convergence among all technologically mature civilizations that lose interest in creating these kinds of simulations. So that’s what the simulation argument itself proves. Then you need some additional arguments and reasons, if you want to conclude that we’re in a simulation, you need to rule out the other 2 alternatives where I think maybe there are plausible arguments for not thinking those alternatives are likely, but they’re not as ironclad as the simulation argument itself. So one reason initially was, well, if I said a specific number, it might give this false sense of precision as if a particular number drops out of this argument, which it doesn’t.

PETER MCCORMACK: (01:01:54 – 01:02:06) So probability instead. There’s a likely probability that this is more likely a simulation rather than a numbered probability. It’s more binary.

PROF. NICK BOSTROM: (01:02:07 – 01:02:08) Sorry, say that again.

PETER MCCORMACK: (01:02:08 – 01:02:21) So rather than say a percentage on it and say it’s a 90% chance or a 50% chance, trying to be that precise, it’s better just to say these are the chances of this and these are the arguments for and against each scenario.

PROF. NICK BOSTROM: (01:02:24 – 01:02:46) Yeah, well, whether it’s better or not, I don’t know. It’s to not misrepresent what the simulation argument actually showed, right? Because the logic of it doesn’t plunk out a specific probability. It imposes this constraint on what you can coherently believe about the future and your position.

PETER MCCORMACK: (01:02:46 – 01:02:46) Yeah.

PROF. NICK BOSTROM: (01:02:47 – 01:03:03) In reality. But then you might have additional hunches or guesses or other arguments that you bring to bear that might then help you narrow that down to a specific number. And presumably there will be a lot of subjective judgment involved in that number.

PETER MCCORMACK: (01:03:03 – 01:03:11) But it’s interesting as a field because we have people now trying to escape the simulation.

PROF. NICK BOSTROM: (01:03:12 – 01:03:17) Yeah. I don’t know whether that’s wise to do, but—

PETER MCCORMACK: (01:03:17 – 01:03:18) Do you think that’s high risk?

PROF. NICK BOSTROM: (01:03:18 – 01:03:25) I mean, it probably doesn’t work at all. If the simulators don’t want us to be able to escape, then certainly it wouldn’t work.

PETER MCCORMACK: (01:03:26 – 01:03:27) Unless it was a test.

PROF. NICK BOSTROM: (01:03:28 – 01:03:28) Yeah.

PETER MCCORMACK: (01:03:28 – 01:03:29) To see who could escape.

PROF. NICK BOSTROM: (01:03:29 – 01:03:33) Well, yeah, if that were the test, then yeah.

Closing Thoughts: A Fretful Optimist

PETER MCCORMACK: (01:03:34 – 01:03:41) Nick, I really, really appreciate your time. I’ve been looking forward to talking to you a long time. Is there anything I haven’t asked you about you wish I had that people would find super interesting?

PROF. NICK BOSTROM: (01:03:42 – 01:03:58) Well, the conversation could go in many different directions from here, I think. There is a lot of stuff to think about in the world and in the current era. So I don’t know, just whatever you are interested in.

PETER MCCORMACK: (01:03:58 – 01:04:20) Yeah. Well, listen, I really appreciate your time. You’ve been very generous. And yeah, look, I think about this a lot, especially the AI. It’s taken over my life in many different ways and it’s almost all entirely positive. And I really hope that we do solve this alignment problem. We don’t have to slow things down because I think there’s so much potential out there.

PROF. NICK BOSTROM: (01:04:21 – 01:04:53) Yeah. I think we should put our best foot forward and then, well, I think we need to get a little bit lucky as well. We’ve always depended to a great extent on luck. Certain things are in human control and we should do those to the best of our ability, but there’s also a degree of, so, people ask me whether I’m an optimist or a pessimist. I tend to say I’m a fretful optimist. But it misses out the third component of my outlook, which is a degree of moderate fatalism as well.

PETER MCCORMACK: (01:04:53 – 01:04:53) Okay.

PROF. NICK BOSTROM: (01:04:54 – 01:05:35) So I think with this alignment problem, for example, it could turn out to be relatively easy, in which case we’ll probably solve it and be fine, or it could turn out to be really hard, in which case we’re maybe doomed no matter what, or it could be intermediate, in which case it might make a big difference the degree to which we get our act together and really, right, so we should, because it might be intermediate, we should really make our best effort. But a lot of the uncertainty about how well things turn out for me is uncertainty about how hard the problem intrinsically turns out to be, as opposed to uncertainty about the degree to which we will make a good effort. Although there’s some of that as well.

PETER MCCORMACK: (01:05:36 – 01:05:39) But as humans, we like solving hard problems. That’s one thing we do like doing.

PROF. NICK BOSTROM: (01:05:41 – 01:06:07) Well, we like it when we succeed at solving hard problems, I guess, but we try. Some hard problems we try and other hard problems are relatively neglected still, I think. Now this problem is becoming less neglected. But we’ll see if the try harder method works.

PETER MCCORMACK: (01:06:08 – 01:06:13) Nick, thank you so much for your time. I really do appreciate it. And just thank you to everyone for listening, and we’ll see you all soon.

Related Posts

• Transcript: How AI Swarms Are Already Going Rogue: Daniel Kokotajlo
• Transcript: Elon Musk & Jensen Huang Speak At America.Gov Event
• Transcript: CMG Interview with Tesla CEO Elon Musk 
• Transcript: UN Security Council AI Hearing w/ Sam Altman, Dario Amodei & Clément Delangue 
• SparX Interview: w/ Paras Chopra – Can Indian IT Services Survive AI? (Transcript)