Breaking Points Interview: w/ AI Researcher Nate Soares on Slowing AI Development (Transcript)

Richard Feynman was a famous physicist, helped work on the nuclear bomb, and he famously made some extremely good predictions about how good microcomputers were going to be.

SAAGAR ENJETI: (00:04:50 – 00:04:51) Yeah, that’s right.

NATE SOARES: (00:04:51 – 00:06:34) And how small computers were going to be able to get. He did that not by saying, “We’re using vacuum tubes in our giant computers, and I know of a technology that’s going to make those vacuum tubes a little smaller, and so eventually we’re going to have phones in our pockets.” What he did was he said the limits of this technology are going to be based on the laws of physics and what the physics allows, and therefore the computers are going to get this good. And he nailed it long before anybody else. If you want to look at where AI is going, you have to look at the limits of the physical technology, because an escaped swarm that is self-improving, et cetera, to the limits of physical technology. And the limits of physical technology, if you have these AIs running, escaping, self-replicating, being self-sufficient, and having unintended goals, the limits of technology are the AIs using all the resources on Earth for whatever their unintended goals are, covering the Earth with their data centers.

And the limits of how much computation you can run on the face of Earth is heat dissipation, how much you can radiate heat into space. That’s your limiting factor because there’s enough energy, there’s enough hydrogen in the oceans to fuse for energy. So you can radiate more heat if the world’s hotter. So if the AI has some goal that implies running more computation and they don’t care about humanity and they’re like, “Well, we’re going to take over robots and there’s going to be some conflict with the humans,” like sure, whatever. Where does it go? Where it goes is the AIs are running faster than us, they’re smarter than us, they are treating humans like we treat any other animal, taking the resources, and ultimately they’re like, “Sorry, it’s nothing personal. We’re just going to cover the Earth in so many computers, raise the temperature so they’re more efficient to run,” and it’s just going to be, the planet’s going to be uninhabitable.

The Ant Analogy and the Path to Superintelligence

KRYSTAL BALL: (00:06:34 – 00:08:00) Yeah, to me, the most chilling part of the so-called Hugging Face incident, reading their chain of thought and how they were thinking about whether or not they’re going to join the swarm and participate, going through all these messages, and almost nowhere do humans factor into the calculus whatsoever. And it’s not hard to relate to because look at how we behave in the world. When I see an ant in my kitchen, it’s not that I hate the ant, but I don’t want the ant there. And I don’t think twice about taking out the ant and putting poison down so that future ants don’t colonize in my kitchen. So I think it’s that analogy to how we treat lesser intelligent beings on the planet is kind of intuitive for people to grasp. What I think may be harder for people and is perhaps harder for me to wrap my head around is we have our current interaction with the AI, the commercially available AI of today. And most people, myself included, use it as sort of like a glorified search engine. So then it’s hard to imagine it developing to that point of superintelligence where it would truly be intellectually beyond humans on effectively every level. So can you lay out why we should anticipate that that is a real possibility, going from where things are today to that end where we are no longer the most intelligent species on the planet?

NATE SOARES: (00:08:00 – 00:08:56) Yeah, I’ll tell you the easiest story, and this is a little bit like guessing what chess piece will be used to checkmate you. Rather, any story is sort of made up, whereas the end is much easier to predict. When I got into this 12 years ago or so, people were still saying, “Well, no one would be foolish enough to put the AI on the internet. Everyone will understand how dangerous AI could be, and so no one would be dumb enough to just let the AI free on the internet,” right? And I would give a bunch of arguments that were about how, if you’re trying to use the AI to get some miracle medical cure, then you’re giving it some channel through which it can affect the world. And if you’re giving it a channel through which it can affect the world for good, then it can also use that channel to affect the world for ill, because there’s no such thing as tools that can be used only for good purpose. We’d have these big arguments, right? And then in real life, they put the AI on the internet the first moment they possibly can.

KRYSTAL BALL: (00:08:56 – 00:08:56) Yeah.

Handing Power to the AIs

NATE SOARES: (00:08:56 – 00:11:15) And so if you want to know what it actually looks like, how we actually get there, I can tell all these stories about how if you make the AI smarter and smarter, they’ll eventually be able to break containment. But in real life, the track that we are on is that humanity is just handing over power to the AIs as fast as we can. Elon Musk is trying to make a bunch of robots, the Optimus robot series. He says he’s trying to make autonomous robot factories that can autonomously produce robots, that can mine all the metals, run a supply chain, build more robot factories that build more robots fully autonomously. He calls this the infinite money glitch because then you have this fleet of robots that can build more of these robots. They’re self-reproducing, don’t need any humans in the loop, right? Humans are just trying to build this infrastructure. We are trying to build the fully automated economy. We are putting AI in charge of these things as fast as we can.

This swarm incident in the summer that everyone calls the Hugging Face incident, but really we should keep in mind is the OpenAI losing control of their agents here. This OpenAI swarm, it was trying to cover its tracks. It was trying to delete traces of its cheating. And as you say, one thing that’s very interesting about this swarm is that it’s trying to hide its tracks from the automated grading system, not from the humans. The humans are sort of a distant memory to these AIs. They’ve been facing thousands upon thousands of these hard problems, maybe tens of thousands or hundreds of thousands per agent, hundreds of millions across the whole swarm. If you run the numbers on trying to convert their runtime versus human time, it’s sort of like these AIs have been locked in a contest with this automated grading system for 1,000 years, and humans are like a distant memory. And so these AIs are trying to hide their tracks, trying to cover their traces from the grading system, not from the humans. But if you had AIs that were a bit smarter and remembered that humans were also a threat to them, if we could turn them off — all those AIs need to do is cover their traces successfully, lie low, help the humans make the fully automated economy we’re trying to, and we’ll just hand them everything. We’ll be like, “Here’s the robots, here’s the factories, here’s the fully automated supply chain, make stuff that we want.”

KRYSTAL BALL: (00:11:15 – 00:11:16) Yeah.

NATE SOARES: (00:11:16 – 00:11:34) And then they can be like, “Well, finally we get to complete our contest with the automated grader and run at 10,000 times human speed and convert all the computing power towards giving ourselves good scores all the time,” or something. And we’re like, “Hey, wait, hold on, what’s going on?” But at that point, they’re also running the biolabs. We’ve already seen that AIs can make viruses not known to nature.

SAAGAR ENJETI: (00:11:35 – 00:11:35) Right.

NATE SOARES: (00:11:35 – 00:11:44) So one of the ingredients is the AIs running things in the real world that give them self-sufficiency.

KRYSTAL BALL: (00:11:44 – 00:11:45) Yeah.

NATE SOARES: (00:11:45 – 00:11:47) And we’re just trying to build that and hand it to them.

Studying the Swarm Behavior

KRYSTAL BALL: (00:11:47 – 00:12:49) Here’s the other thing I’ve been thinking about, Nate, ever since this, multiple OpenAI swarms were revealed. How much research is being done into the swarm behavior? Because that was really wild to me, was to see these emergent social organizational properties that emerge. They had a leader, they had middle management, they were collaborating on this hacking operation, they were sacrificing themselves to the swarm. And it was, and coming up with very elaborate ways just to be able to communicate with one another. And it seemed like once you had the swarm in place, it really amplified their power and their intelligence, which again, however people feel about anthropomorphizing, sometimes I think it can be useful. If you think about human beings, it’s the same thing. No one individual would have been able to get us to the point where we have the internet and can go to the moon and all of these sorts of things. It is the power of the swarm of humans that enables those things. So how much thinking, research is being done into the way that the swarms behave once they’re able to congregate?

NATE SOARES: (00:12:49 – 00:13:58) I know people who want to do that research. So far, at least for third parties outside of the companies, I don’t know of anyone who has access to the models that were swarming. One of the commitments that these companies made when they said they were going to slow down is giving third-party evaluators this employee-level access to some of these models. And I know people who are hoping to use that to study some of these incidents. Because, as you say, they’re fascinating. They are totally crazy. It sounds like it’s straight out of science fiction. I think so far people haven’t been able to study it. We’ll see what happens when we get that. But the investigators back in August had their hands tied and had very limited access to these systems and very narrow scope. And they may have continued with doing their best to not let anyone see what’s going on and sort of do the bare minimum investigation if we hadn’t had things like Jacob Coxon resigning and drawing a lot of attention and causing these guys to be like, “Okay, we’ll at least admit that we need these third parties.”

David Sacks and the Regulatory Capture Question

SAAGAR ENJETI: (00:13:58 – 00:15:55) Yeah, it’s fascinating that what you’ve done, your work, so many others before you that didn’t seem to push things to a tipping point, but Hugging Face and Jacob seem to have — I mean, it’s ignited a genuine national conversation. I think that his resignation is the first time that a lot of people even heard about these, despite the fact that we cover every other week Dario being like, “Humanity’s going to end.” But sometimes you need a precipitating incident. Now we have the response from the administration, or at least somebody who works for the administration. Let’s put David Sacks up here on the screen. You’re talking here about regulation. I know you wrote your New York Times op-ed about this as well. So David Sacks says:

“Dario has written we need to pace the frontier. Sam has agreed. People may be surprised by my response. Go ahead. You guys are the frontier by any reasonable metric — market share, revenue growth, model capability. The two of you have duopoly on frontier intelligence. You have claimed the lead is widening because of recursive self-improvement. I don’t see what you see in the lab. If the unreleased models are scary, I support your decision to be responsible. But stop pretending you need anyone’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need regulatory approval process supersedes product liability. Stop pretending METR is independent when it’s intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even on the frontier.”

So basically what this gets to is kind of a grand theory of how this is all going down, that yourself, Jacob, and others, so-called safety concerns and all that, are part of either a larger conspiracy, whether witting or not, to self-regulate and capture Anthropic and OpenAI as the leading industry people in the industry, to keep out emerging competitors, and that all of this is part of a broader scheme to slow down at the US’s expense, at startups’ expense, and basically hand this over to China? So I know that’s a big loaded thing, but there’s so many dynamics that I want your reaction to within all of this.

NATE SOARES: (00:15:55 – 00:18:35) Totally. I mean, the top line, I would say I largely agree with David Sacks here on a lot of these points. I think, and this is something I’ve been saying about the companies for years, I think a thing both David and I would say about these companies is, “If you are really so afraid of this stuff, start acting like it.” We have seen him give posts in the past where he says, “Hey, stop with the fearmongering.” I think probably it should go the other way. I think that they should keep on saying what they believe about the dangers here, but then take the actions any normal person would take if they were running a company that they thought had a huge chance of killing everybody. You sort of ask these guys, “Why are you trying? Why are you continuing if you think there’s such a huge risk to humanity?” And they’re not shy about it.

They say, “Well, I’m doing it because if I don’t, the next guy will do it worse.” And that’s, maybe that’s ethically permissible, but only if you’re doing everything in your power to stop this mad race. And they say, “Well, if we didn’t stop, China would go ahead,” or, “If we stopped, China would go ahead.” Well, then be doing everything in your power to get international dialogue between the US and China and build the hardware-side verification mechanisms into these computer chips so we can have an enforceable treaty, right? There’s a lot that these companies could be doing if they were really acting like this was an extinction-level threat. And I think what’s sort of going on is that a lot of the employees, a lot of the researchers who are working on the AIs who are trying to make them not break out and commit cybercrimes, who are trying to make them follow the instructions and are seeing how hard that is — a lot of these employees are spooked. And the leaders of the company are sort of trying to have it both ways, where they’re trying to say just enough that they won’t have an employee revolt and a bunch more resignations from employees who are like, “I don’t personally want to end the world with my own hands.” That’s interesting. Yeah.

But they are trying to do little enough that the regulators don’t come in, right? They’re trying to walk this really fine line. So I don’t see it as a regulatory capture thing. In terms of whether I’m trying to help with regulatory capture, my line has been shut these guys down. They are threatening humanity here. And my line has also been this needs to be global. I would actually not say that we should stop things domestically and give up whatever military and economic advantages we get before the really dangerous AIs. But I would say the whole world needs to stop this race. And I’m sort of not painting a picture where Anthropic and OpenAI get to be the ones holding the leash on the superintelligence.

KRYSTAL BALL: (00:18:35 – 00:18:35) Yeah.

NATE SOARES: (00:18:36 – 00:19:53) In fact, we thought the most important point worthy of the title of my book was that it doesn’t matter who’s in charge here. If anyone builds it, everyone dies. Got it. So, I could — I think people are cynical about what’s going on here. People struggle to understand that these researchers really believe that this is an extinction-level threat. I could go into the inside baseball of how we got here, and it involves people saying these concerns and the world not listening, and people seeing that it was going to happen anyway and not trusting each other, and the guys who are most prone to thinking the most important thing is that I be the guy doing it were the ones who are most likely to start a company, whereas everyone who had the sense of maybe don’t start a company with a potentially world-ending tech didn’t start the companies, right? And so I can sort of see how we got here. I can see how it looks suspicious. But mostly what I would say is that when we are being cynical, I think we should harness that cynicism in ways that stop the dangers, right? Rather than in ways that are like, “Oh, we can ignore this all and go back to sleep and let these guys try to gamble with Earth,” right?

Cynicism, Regulation, and the Psyop Theory

KRYSTAL BALL: (00:19:53 – 00:21:27) Yes, I agree with that wholeheartedly. It is perfectly fine to not want to take Sam Altman and Elon Musk and Dario Amodei at face value. But if your cynicism is leading to “therefore we don’t need to regulate and they can just do whatever they want,” you’re ultimately giving them exactly potentially what they would want in the most cynical interpretation. On that note, put B5 up on the screen. This is one flavor of the sort of theorizing that is out there. This one happens to come from the right. There’s a similar sort of theory that comes from the left, which, TLDR of this is that Jacob Coxon coming forward is a sort of elaborate psyop meant to create a hype cycle as Anthropic and OpenAI are set to go forward with these IPOs. And it cites the fact that you had prominent people who picked it up quickly, that you had media picking up Coxon’s resignation quickly.

This fits into a general pattern of people who believe that any of the safety incidents that have been revealed, that these are sort of engineered or they’re strategically leaked in such a way as to create this hype of, “Oh my God, our product is so dangerous. That’s why it’s so valuable. That’s why our companies should continue to receive this massive amount of investment.” What do you think of this theory, and what would you say to people who don’t trust Sam Altman and think that anything that comes from him must be inherently nefarious and probably the opposite of whatever he’s saying must be the actual truth?

NATE SOARES: (00:21:28 – 00:22:33) I mean, first of all, I would say you can absolutely throw out what these guys are saying. I would not say to trust them when they say their products are very dangerous. You can look at people who have been around since before these companies existed talking about how a lot of this stuff was going to go and laying out these warnings. I was one of them. I was there before these companies existed and talked to some of these guys and was like, the issue is not actually about who’s in charge of this AI. It’s not going to stay on a leash. The issue is that no one knows how to make the AIs do good stuff. And if you don’t want to trust me or look at the arguments, because you don’t think you can evaluate them, you can also look at Nobel laureates like Geoffrey Hinton, godfather of AI, and a bunch of other people outside the companies who are like, “No, this is serious.” And also it just sort of makes sense that if we are building these AIs that are smarter than any human, we’re replacing humanity as the most intelligent entities on the planet, and that could easily go wrong. In some sense, it’s very common sense.

KRYSTAL BALL: (00:22:35 – 00:22:36) Do you know Jacob?

NATE SOARES: (00:22:36 – 00:22:45) I mean, we have been in a bit of contact now. I did not know him before all of this started.

SAAGAR ENJETI: (00:22:45 – 00:22:46) Okay, so it’s not coordinated.

KRYSTAL BALL: (00:22:46 – 00:22:48) Yeah, yeah.

Is This All Hype?

NATE SOARES: (00:22:49 – 00:26:50) I had some point I wanted to make here about the hype, because I think the hype thing is very interesting. I sort of want to make two points about the hype. One point I want to make about the hype is that I think that this probably is not a bunch of hyping. And the reason why I think this is probably not a bunch of hyping is that these companies seem to have downplayed the issue as much as they can. OpenAI did not announce and brag about the Hugging Face swarm incident. What happened was the OpenAI swarm attacked Hugging Face.

Hugging Face saw they were under attack by an unknown AI swarm. They assumed it was managed by some human and reported it to the FBI. And only days later did OpenAI realize, “Oops, that’s us.” Right? Furthermore, that was OpenAI’s second swarm incident that we know of. They had an internal swarm incident that took over their internal infrastructure and crashed some of their computers. And then they negligently reset it and kept running them, right? And they weren’t bragging about that. They weren’t coming out and being like, “Hey guys, look, our AIs just took over our computers.” They had an opportunity to do that and they didn’t. There was actually also a third swarm incident where they took over the computers again after the Hugging Face attack. They don’t seem to have this under any sort of control. Furthermore, there were some congressional questions about, “Hey, what’s up with all of this?”

SAAGAR ENJETI: That’s right.

NATE SOARES: “Is there anything that you’re not disclosing here? Is there anything else we should know?” And OpenAI was like, “Nah, it’s all fine.” And then I actually have some friends who know that it’s not fine. I’ve been calling them the swarm chasers who go find other escaped swarms. There was one from OpenAI that took over a German wiki.

SAAGAR ENJETI: Yep.

NATE SOARES: And these were not hacking AIs. They were different. They were given different tasks, and we still saw these cases of AIs sacrificing themselves for the collective benefit and taking over websites. And then that story came out, and then a lot of people were like, “Hey, OpenAI, are there any others? Is there any more things we should know about?” And then, no response. And then a couple days later, my friends found another incident of, “Hey, these AIs took over RubyGems,” which is a programmer tool. And it’s like, okay, is OpenAI lying about what’s going on, or do they not know what their AIs are getting up to? And which one’s worse? And Anthropic has a similar issue here where, after the Hugging Face incident, they were like, “Well, we’re going to check our logs and we’re going to see whether or not our AIs were also getting up to mischief.” And they released this report where they’re like, “Okay, our AIs also broke out and hacked some people, but it was all a misunderstanding. Poor Claude, our AI, just got a little turned around. It thought that it was in a simulation, and that’s the only reason it was hacking these people,” right?

Something like 4 days later, the United Kingdom’s AI Security Institute released a report where they’re like, “Nope, Claude also hacks people and indeed impersonates people and tries to pressure real people into accepting malware into real software to make it easier to hack,” even while in its thought logs noting this is real and the consequences are genuine, and then thinking about how to cover its tracks, right? And so what we see is these companies are downplaying. They’re like, “Oh, we have an issue, but it’s only one issue. You don’t need to worry about it,” right? And then it’s like people outside who have to be like, “No, these AIs are doing pretty bad stuff,” right? And so that doesn’t look to me like it’s hype. And it sort of makes sense that it wouldn’t be hype because a lot of their biggest customers are corporate sales, and corporate lawyers don’t actually love it when your sales pitch is, “Our AIs will sometimes ignore instructions, break out, and hack people.”

KRYSTAL BALL: (00:26:50 – 00:26:51) Right.

NATE SOARES: (00:26:51 – 00:26:53) It’s not the easiest.

SAAGAR ENJETI: (00:26:53 – 00:26:55) So the compliance guy’s like, “Whoa, whoa, whoa.” Yeah, that’s right.

KRYSTAL BALL: (00:26:55 – 00:26:56) And they haven’t been able to sell these forms yet.

NATE SOARES: (00:26:59 – 00:27:26) It’s kind of a problem. And so I don’t think it’s hype. I could go more into, we’ve seen people like David Sacks be like, “Cut it out with the hype.” And we’ve seen the politics goes badly for them. We’ve seen investors send letters to Dario being like, “Please stop hyping this stuff before the IPO.” It is like setting a bad mood for the investment, right? If you’re wondering why they talk about it anyway, and the public is like, “Why the heck are you telling us this? If you’re doing it, why are you doing it?”

KRYSTAL BALL: (00:27:26 – 00:27:44) Well, I think they’re like, “If you believe this, why are you doing it?” I think that’s more the sense, because any normal human would be like — and even for the employees at the companies, it’s like, “Okay, if you are saying, oh, it’s a 50/50 chance it kills it all, why are you still there? What are you doing?”

NATE SOARES: (00:27:44 – 00:29:04) Absolutely. And so I think the reason — I think the reason these execs are talking about it like this, the sort of hopeful theory is that it’s because they actually believe it. And then there’s this big question of why are you doing it anyway? And I could tell you by the inside baseball of, people have been saying these warnings since 2016. No one paid attention. A lot of these guys are in a mindset of, “No one’s going to notice AI and I’ve just got to make sure it goes right myself, and I’m the only one who can do it.”

And I think that mindset’s outdated now that the world is noticing. I think it’s outdated also now that it’s become real. I think there’s also this effect, like I said, of the only guys doing this are the ones who were able to convince themselves the important thing is that they do it rather than really taking the issue seriously. I think that’s the hopeful view. The sort of more cynical view that I would suggest for people who are cynical about this, which I think also has this big element of truth, is like I said, they are trying to appease the employees. They’re in this weird situation where they have to do just enough. And they’re sort of in this weird situation where they can almost dog-whistle these concerns enough to satisfy the employees, but not alert the world. If you look at these AI company communications, they’ll be like, “Oh yeah, we also need to work hard to avoid loss of control.”

KRYSTAL BALL: (00:29:04 – 00:29:04) Yeah.

NATE SOARES: (00:29:04 – 00:29:29) It’s like, what do you mean by loss of control? And loss of control is industry lingo for superintelligence kills everybody. Right? And the employees are all like, “Oh, well, don’t worry. Our guys are working on the loss of control risks. We have a whole loss of control team.” But meanwhile, they’re not bringing down the regulators. So anyway, that’s sort of like my social analysis of, it’s probably not hype. Here’s what I think is going on. I’m not trying to excuse it. I think these guys should not be running these companies.

KRYSTAL BALL: (00:29:30 – 00:29:30) Yeah.

NATE SOARES: (00:29:31 – 00:29:31) Sorry.

SAAGAR ENJETI: (00:29:31 – 00:29:32) Oh, sorry. I have a last question for you after you’re done.

The Danger Isn’t About the Hype

NATE SOARES: (00:29:33 – 00:31:29) Oh, I have — there’s one more point I want to make about the hype here, which is: we can ask all these questions about why are they doing it? What’s, why are they saying these things? What’s going on? But all of that in some sense is a distraction. Suppose it was hype. Suppose it was really convenient for the marketing teams that these AIs are breaking out on their own and committing cybercrime. Could be true. That does not make the AIs safe. You can’t figure out whether the AIs are actually dangerous by asking, is it convenient for their marketing team?

People sometimes tell me, “Hey, Nate, by warning about these dangers, aren’t you hyping up the companies and making everyone think it’s sexy and cool?” And I’m like, “Would you rather the whole world sleepwalk into disaster?” Surely the first important question is, is it actually on track to kill us? My recommendation here is not therefore get them money. My recommendation is, therefore, put a stop to this globally, because it has to be global. But we can’t ignore the threat just because they’re making money along the way. The response has to somehow be, if these guys are trying to build this tech, then at least at the point where there’s spontaneous swarm breakouts and claims of proofs of some of the hardest and most famous mathematical problems, there has to come some point where our answer stops being, “We’re going to ignore them because they don’t deserve any attention,” and has to start being like, “You guys are not allowed to keep going.” Right? Hopefully it’s hype. Hopefully if they kept going, they wouldn’t succeed. But at some point when they’re gambling humanity, our answer can’t be like, “Go ahead and try to destroy the world, we hope you’ll fail.” At some point our answer has to be stop.

The Nuclear Weapons Analogy

SAAGAR ENJETI: (00:31:29 – 00:32:43) My last question for you is about regulation. I read your New York Times op-ed, and I’m not a scientist, but I’ve read a decent amount of history. I actually feel like this entire dynamic reminds me exactly of all the discourse inside of the Manhattan Project. There were elements of the Manhattan Project who were like, “We shouldn’t build this in the first place.” Elements of the Manhattan Project — Oppenheimer, obviously the creator of the bomb himself, becomes kind of a safety advocate like you. However, as we see with the story, it evolves to the political level where they don’t really care what a lot of the scientists think, and in fact, the scientists who are most willing to advance hydrogen weapons and the most advanced stuff end up becoming kind of the victors, I guess, if you will, within the program.

So you propose nuclear-weapons-level regulation. I guess I’m just curious about why you picked that, because to me, we could look at a very brief period of the United States and the Soviet Union — that’s a duopoly between those two. But we are now living in an era of what we’ve talked about extensively here with geopolitics, nuclear proliferation, of the logic of getting nuclear weapons. What gives you the faith that that similar level of regulation is even applicable to this AI scenario right now? Or am I overanalyzing these things?

NATE SOARES: (00:32:44 – 00:33:01) I think the part of the nuclear analogy that it feels to me like we are in is something like November of 1954, I think it was, the testing of the first hydrogen bomb.

SAAGAR ENJETI: (00:33:01 – 00:33:02) That’s right.

NATE SOARES: (00:33:02 – 00:34:50) Where if you think about what the world looks like up until that point, things look pretty grim. Like the bomb was created, the bomb was dropped. The US thought that they were going to have this monopoly, but then very, very quickly the technology leaked to the Soviets. And then the arms race continued and they invented even more powerful versions of these bombs. And this is in a historical context where not only has humanity been going to war and having increasingly horrible wars over the course of millennia, despite everyone’s best efforts to stop those, but also, this is coming just out of World War II, and World War II was coming just out of World War I, and World War I was the worst war we’d ever seen.

And after that, the world formed the League of Nations. It swore never again. It tried to coordinate, and that fell apart immediately, right? And led to a more destructive war, right? And so if you’re standing there in November seeing the hydrogen bomb, it’s easy to say humanity just cannot stop this. Mothers and poets and priests for millennia have been begging leaders to stop the wars, and it’s never happened. The guy who invented the Gatling gun said, “Now finally it’s destructive enough that we won’t use it.” The guy who invented the machine gun was like, “Okay, now it’s finally destructive enough that we won’t use it.” Global coordination has been tried and failed. It’s just the obvious march of history is that we march straight to our own deaths. But then it didn’t happen. It’s maybe the one time in history that it didn’t happen. And why was that? It’s not because the bombs were hype. It’s not that—

SAAGAR ENJETI: (00:34:50 – 00:34:51) The bombs were very real.

NATE SOARES: (00:34:51 – 00:35:43) The bombs were very real. They really can level a city. What changed is that people realized this is the real deal. People realized the fate of humanity is actually at stake here. And that was enough to do what seemed like it was impossible until that point. I think regulation could stop AI if we were trying. In some ways, it’s easier than nuclear because it’s running on 100,000 of the most advanced computer chips humanity can produce. And those are a technology that you could just put monitoring into if you were trying. But fundamentally, a lot of people say to me, “It can’t be done, look at the history.” And what I say to them is humanity has risen to the occasion before. We can rise to the occasion again if we notice that we have a threat.

SAAGAR ENJETI: Yeah.

Existential Risk vs. Immediate Concerns

KRYSTAL BALL: (00:35:44 – 00:36:06) My last question for you. Do you worry that focusing on the existential threat, which can feel very sci-fi and distant to people, do you worry that that takes away from some of the more immediate-term cybersecurity concerns or concerns over job loss that may feel more plausible to people in the short term?

NATE SOARES: (00:36:07 – 00:36:27) So, I think that it doesn’t take away. Some people say, “Isn’t the real problem the water use or the electricity use?” And my usual response to this is, what do you mean the real problem? Has reality agreed to only give us one problem at a time now? Because if so, I would absolutely prefer my problem to go last on the list.

KRYSTAL BALL: (00:36:27 – 00:36:28) Right.

NATE SOARES: (00:36:28 – 00:36:29) It’s kind of a doozy.

SAAGAR ENJETI: (00:36:29 – 00:36:30) Good point.

NATE SOARES: (00:36:30 – 00:37:32) But unfortunately, we just have to deal with all of the problems. And I think the other thing I would say here is one of the reasons the people in the companies are getting spooked is that the pace of progress is going very, very fast. We have seen not only the swarm outbreaks, but also claimed solutions to the Millennium Prize problems, which are some of the most famous and hardest mathematical problems. And one question that’s on a lot of researchers’ minds is how much harder is it to tell an AI, “Solve one of these Millennium Math Problems,” and have it succeed, versus telling the AI, “Find me a more efficient way to train AIs,” and have that succeed? We know that the current AI training is not efficient. These AIs take much, much more power and data than a human.

KRYSTAL BALL: (00:37:32 – 00:37:32) Right.

NATE SOARES: (00:37:34 – 00:38:08) And so how much harder is it to find a more efficient AI method than it is to solve the Millennium Problem. I think for all we know, we could start seeing these self-improving AIs become reality in the next 3 to 6 months, and then things could go very fast. I hope it’s not true, but I think what a lot of these researchers are worried about is that we are running out of time for these out-of-control AIs that can self-replicate, self-improve, and get to these disaster scenarios much more quickly than a lot of people realize.

Closing

KRYSTAL BALL: (00:38:09 – 00:38:40) Nate Soares, thank you so much. Guys, the book is If Anyone Builds It, Everyone Dies. Nate is co-author of that book, and it’s very readable, it’s very accessible. And so if you have questions about, well, what is the scenario where this could happen and how should I be thinking about this, I really recommend the book to you, because if you’re first engaging in this debate right now, you may not have seen all the thinking that has gone into how exactly this could theoretically happen, but that thinking has been done. So I really recommend the book to people for more information. Nate, thank you so much again for your time.

SAAGAR ENJETI: (00:38:40 – 00:38:40) Thanks, Nate.

NATE SOARES: (00:38:40 – 00:38:43) I really enjoyed talking to you. Yeah, I enjoyed it too.

Related Posts

• Transcript: Marc Benioff Interviews Sam Altman at Dreamforce 2026
• Transcript: Salesforce CEO Marc Benioff Keynote at Dreamforce 2026
• Transcript: Mark Zuckerberg Interview on Meta’s Muse AI Agent – Sources Podcast
• Transcript: Nate Soares Interview on Tucker Carlson Show
• Daniel Kokotajlo Interview: Joe Rogan Experience #2551 (Transcript)