Transcript: The Future of U.S. AI Leadership with Dario Amodei of Anthropic

They’re called scaling laws or the scaling hypothesis today.

The basic hypothesis is simple. It says that—and it’s really a remarkable thing, and I can’t overemphasize how unlikely it seemed at the time—if you take more computation and more data to train AI systems with relatively simple algorithms, they get better at all kinds of cognitive tasks across the board. And we were measuring these trends back when models cost $1,000 or $10,000 to train.

So that’s a kind of an academic grant budget level. And we forecast that these trends would continue even when models cost $100 million, a billion, $10 billion to train, which now we’re getting to. And indeed, that if the quality of the models and their level of intelligence continued, they would have huge implications for the economy.

It was even the first time we realized that they would likely have very serious national security implications. We generally felt that the leadership at OpenAI was on board with this general scaling hypothesis, although many people inside and outside were not. But the second realization we had was that if the technology was going to have this level of significance, we really needed to do a good job of building it.

We really needed to get it right. In particular, on one hand, these models are very unpredictable. They’re inherently statistical systems.

One thing I often say is we grow them more than we build them. They’re like a child’s brain developing. So controlling them, making them reliable is very difficult.

The process of training them is not straightforward. So just from a system safety perspective, making these things predictable and safe is very important. And then, of course, there’s the use of them, the use of them by people, the use of them by nation states, the effect that they have when companies deploy them.

And so we really felt like we needed to build this technology in absolutely the right way. OpenAI, a bit as you’ve alluded to, was founded with some claims about that they would do exactly this. But for a number of reasons, which I won’t get into in detail, we didn’t feel that the leadership there was taking these things seriously.

And so we decided to go off and do this on our own. And the last four years have actually been a kind of almost a side-by-side experiment of what happens when you try and do things one way and what happens when you try and do things the other way and how it has played out.

Anthropic’s Commitment to Safety and Responsibility

So I’ll give a few examples of how we’ve really, I think, displayed a commitment to these ideas.

One is we invested very early in the science of what is called mechanistic interpretability, which is looking inside the AI models and trying to understand exactly why they do what they do. One of our seven co-founders, Chris Ola, is the founder of the field of mechanistic interpretability. This had no commercial value or at least no commercial value for the first four years that we worked on it.

It’s just starting to be a little bit in the distance, but nevertheless, we had a team working on this the whole time in the presence of fierce commercial competition because we believe that understanding what is going on inside these models is a public good that benefits everyone, and we published all of our work on it so others could benefit from it as well.

I think another example is we came up with this idea of constitutional AI, which is training AI systems to follow a set of principles, instead of training them from data or from mass data or human feedback. This allows you to get up in front of Congress and say these are the principles according to which we trained our model.

When we first came to when we had our first product, our first version of Claude, which is our model, we actually delayed the release of that model roughly six months because this was such a new technology that we just weren’t sure of the safety properties. We weren’t sure we wanted to be the ones to kind of kick off a race. This was just before ChatGPT, so we arguably had the opportunity to seize the ChatGPT moment, and we chose to release a little later, which I think had real commercial consequences but set the culture of the company.

A final example I would give is we were the first to have something called responsible scaling policy. So, what this does is it measures categories of risk of models as they scale, and we have to take increasingly strict security and deployment measures as we meet these points, and so we were the first one to release this, the first one to commit to it, and then within a few months of when we did, the other companies all followed suit, and so we were able to set an example for the ecosystem.

When I look at what other companies have done, we’ve often led the way on these issues and often caused the other companies to follow us. Not always. Sometimes they do something great and we follow them, but I think there’s been a good history of us sticking to our commitments, and I would contrast that with what we’ve seen from some of the other companies in their behavior. We now have several years of history, and so far, fingers crossed, I think our commitments have held up pretty well.

AI Safety Levels and Responsible Scaling

MICHAEL FROMAN: I want to talk about both the risks and the opportunities that you’ve cited around AI, but since you mentioned the responsible scaling issue, let’s go back to that. We’re at level two now.

DARIO AMODEI: Yeah.

MICHAEL FROMAN: And at what level is it existential? How will we know when we hit level three? And if you hit level three, can you go backwards, or does it only get worse?

DARIO AMODEI: The way our responsible scaling policy is set up is we basically said—and the analogy was to bio-safety levels, so the bio-safety level system is like these are how dangerous various pathogens are, and so we said, let’s have AI safety levels.

And so AI safety level two is the level we’re currently at, and that’s systems that are powerful, but the risks they pose are comparable to the risks that other kinds of technology pose. ASL-3, which actually I think our models are starting to approach. The last model we released, we said this model isn’t ASL-3 yet, but it’s getting there.

ASL-3 is characterized, and we focus very much on the national security side, very kind of serious risks that are out of proportion to the risks that normal technologies have. So an ASL-3 model is designed as one that could allow you in the areas of, say, chemical, biological, or radiological weapons, could allow an unskilled person, simply by talking to the model and following its instructions, to do things that you would have to have, say, a PhD in virology to do today.

So once that is possible, if those risks aren’t mitigated, then that would enhance the number of people in the world who are able to do these highly destructive things from, say, in the 10,000s today to in the tens of millions once the models are available.

And so when the models are capable of this, we have to put in mitigation so that the models are not willing to actually provide this information and security restrictions so that the models won’t be stolen. And I think we’re approaching that. We may actually hit that this year.

And we believe we have a story for how to deploy those kinds of models safely by removing their ability to do this very narrow range of dangerous tasks without compromising their commercial viability.

MICHAEL FROMAN: So this is a fairly narrow set of tasks, as you say, you’re just going to prevent the model from answering those questions.

DARIO AMODEI: Yeah, prevent the model from engaging in those kinds of tasks, which is not straightforward, right? But you can say, “I’m taking a virology class at Stanford University. I’m working on my coursework. Can you tell me how to make this particular plasmid?” And so the model has to be smart enough to not fall for that.

MICHAEL FROMAN: And say, “Hey, actually, that isn’t the kind of thing you would have… like a bioterrorist. You sound like you have bad intent.”

DARIO AMODEI: Yeah.

MICHAEL FROMAN: But it’s sort of limited to your own imagination, our own imagination as to what all the back acting could be. There are a lot of things that we may not anticipate beyond those four categories.

DARIO AMODEI: I mean, I think this is an issue that just as every time we release a new model, there are positive applications for it that people find that we weren’t expecting. I expect there will also be negative applications. We always monitor the models for different use cases in order to discover this so that we have a continuous process where we don’t get taken by surprise.

If we’re worried that someone will do something evil with Model 6, hopefully some early signs of that can be seen in Model 5 and we monitor it. But this is a fundamental problem of the models. You don’t really know what they’re capable of. You don’t truly know what they’re capable of until they’re deployed to a million people.

You can test ahead of time. You can have your researchers bash against them. You can even have the government. We collaborate with the government AIS guys, test them. But the hard truth is that there’s no way to be sure.

They’re not like code where you can do formal verification. What they can do is unpredictable. It’s just like, if I think of you or me instead of the model, if I’m like the quality assurance engineer for me or you, can I give a guarantee that there’s a particular kind of bad behavior you are logically not capable of will never happen? People don’t work that way.

The Opportunities of AI

MICHAEL FROMAN: Let’s talk about the opportunities, the upside opportunities.

DARIO AMODEI: Absolutely.

MICHAEL FROMAN: End of last year, you wrote an essay, “Machines of Loving Grace,” that talks about some of the upside, how one could achieve a decade’s worth of progress in biology, for example, in a year, how the machines were going to be as smart as all the Nobel Prize winners, which probably depresses some of them. Tell us the upside. Tell us your best case scenario as to what AI is going to produce.

DARIO AMODEI: So, I’d go back by starting with the exponential. If we go back to 2019, the models were barely able to give a coherent sentence or a coherent paragraph. People like me, of course, thought that was an amazing accomplishment that models were not capable of. And we had these predictions that five years from now, the models are going to be generating billions of dollars of revenue. They’re going to be helping us code.

We can talk to them like they’re human beings. They’ll know as much as human beings do. And there were all these unprincipled objections of why that couldn’t happen.

The same exponential trends, the same arguments that predicted that predict that if we go forward another two years, three years, maybe four years, we will get to all of this. We will get to models that are as intelligent as Nobel Prize winners across a whole bunch of fields.

You won’t just chat with them. They’ll be able to do anything you can do on a computer, basically any remote work that humans do, any modality, being able to do tasks that take days, weeks, months. The kind of evocative phrase that I used for it in “Machines of Love and Grace” was, it’s like having a country of geniuses in a data center. Like a country of genius remote workers, which they can’t do everything, right? There are restrictions in the physical world.

And I think this still sounds crazy to many people, but look back on previous exponential trends, look at the early days of the internet and how wild the predictions seemed and what actually came to pass. I’m not sure of this. I would say I’m maybe 70 or 80% confidence.

The Future of AI and Its Impact

MICHAEL FROMAN: You know, it could very well be that the technology stops where it is or stops in a few months and the essays that I’ve written and things I’ve said in events like this, people spend the next 10 years laughing at me, but that would not be my bet. Let’s just build on that one, because on the issue of jobs and the impact that AI is likely to have on employment, there’s a pretty big debate. Where are you on the spectrum? But before I get there, how long will it take for AI, let’s say, to replace the head of a think tank? I’m asking for a friend.

DARIO AMODEI: Actually, we won’t get to that one. Where are you on the spectrum of everyone’s going to be able to do some really cool things and they’re going to be able to do so many more things than they’re able to do now versus everyone’s going to be sitting on their sofa collecting UBI?

MICHAEL FROMAN: You can also answer the think tank question if you’d like.

AI’s Potential Impact on Health and Employment

DARIO AMODEI: So I mean, I guess I didn’t, I kind of ended my answer to the last question without saying all the great things that will happen. So honestly, the thing that makes me most optimistic before I get to jobs is things in the biological sciences, biology, health, neuroscience.

If we look at what’s happened in biology in the last 100 years, what we’ve solved are simple diseases. Solving viral and bacterial diseases is actually relatively easy because it’s the equivalent of repelling a foreign invader in your body. Dealing with things like cancer, Alzheimer’s, schizophrenia, major depression, these are system level diseases.

If we can solve these with AI at a baseline, regardless of kind of the job situation, we will have a much better world and I think we will even, if we get to the mental illness side of it, have a world where it is at least easier for people to find meaning. So I’m very optimistic about that.

But now getting to kind of the job side of this, I do have a fair amount of concern about this.

On one hand, I think comparative advantage is a very powerful tool. If I look at coding, programming, which is one area where AI is making the most progress, what we are finding is we are not far from the world. I think we’ll be there in three to six months where AI is writing 90% of the code.

And then in 12 months, we may be in a world where AI is writing essentially all of the code. But the programmer still needs to specify what are the conditions of what you’re doing, what is the overall app you’re trying to make, what’s the overall design decision, how do we collaborate with other code that’s been written, how do we have some common sense on whether this is a secure design or an insecure design. So as long as there are these small pieces that a programmer, a human programmer needs to do, the AI isn’t good at, I think human productivity will actually be enhanced.

But on the other hand, I think that eventually all those little islands will get picked off by AI systems. And then we will eventually reach the point where the AIs can do everything that humans can. And I think that will happen in every industry.

I think it’s actually better that it happens to all of us than that it happens, that it kind of picks people randomly. I actually think the most societally divisive outcome is if randomly 50% of the jobs are suddenly done by AI. Because what that means, the societal message is we’re picking half, we’re randomly picking half of people and saying, you are useless, you are devalued, you are unnecessary.

MICHAEL FROMAN: And instead we’re going to say you’re all useless.

DARIO AMODEI: Well we’re all going to have to have that conversation, right? Like we’re going to have to look at what is technologically possible and say, we need to think about usefulness and uselessness in a different way than we have before, right? Our current way of thinking has not been tenable. I don’t know what the solution is, but it’s got to be different than we’re all useless, right? We’re all useless is the nihilistic answer.

We’re not going to get anywhere with that answer. We’re going to have to come up with something else.

MICHAEL FROMAN: That’s not a very optimistic picture, is what it is.

DARIO AMODEI: I actually, I would actually challenge that. You know, I think about a lot of the things that I do. You know, I spend a lot of time, for example, swimming.

I spend time playing video games. I look at like human chess champions. You know, you might think when Deep Blue beat Kasparov, and that was almost 30 years ago, that after that it would be like chess would be seen as a pointless activity.

But exactly the opposite has happened. Human chess champions like Magnus Carlsen are celebrities. I think he’s even like a fashion model.

He’s like this kind of hero. So I think there’s something there where we can build a world where human life is meaningful and humans, perhaps with the help of AIs, perhaps working with AIs, build really great things. So I am actually not that pessimistic, but if we handle it wrongly, I think there’s maybe not that much room for error.

DeepSeek and the AI Arms Race

MICHAEL FROMAN: A couple of months ago, we had DeepSeek being released. In this town, there was a fair degree of panic, I would say, around that. People talked about it as a Sputnik moment.

Was it a Sputnik moment? And what does it teach us about whether those scaling rules that you laid out about needing more compute, more data, better algorithms, whether that’s, those rules still apply or whether there are some shortcuts?

DARIO AMODEI: So DeepSeek, I think, actually was, rather than refuting the scaling laws, I think DeepSeek was actually an example of the scaling laws. So two dynamics, I had a post about this, but two dynamics are going on at the same time.

One is that the cost of producing a given level of model intelligence is falling, roughly by about 4X a year. This is because we are getting better and better at algorithmically producing the same results with less cost. In other words, we’re shifting the curve.

You can get, for a year later, as good a model as you could get a year ago, spending 4X, you can get a 4X better model by spending the same amount. But what that means economically is that whatever economic value the model of a given intelligence has, the fact that you can make it for 4X cheaper means we make a lot more of it. And in fact, provides additional incentive to spend more money to produce smarter models which have higher economic value.

And so even as the cost of producing a given level of intelligence has gone down, the amount we’re willing to spend has gone up. In fact, has gone up fast. Something like 10X a year, despite that 4X a year increase.

That’s been eaten up and more by just society, the economy wants more intelligence. It wants more intelligent models. So that is kind of the backdrop for DeepSeek.

And DeepSeek was literally just another data point on the cost reduction curve. It was nothing unusual. It wasn’t like these U.S. companies are spending billions and DeepSeek did it for a few million.

The costs were not out of line. They spent, yes, a few million on the model, so what U.S. companies spend is not out of line with that. They, like us, spent billions on all the R&D and effort around the model.

If you look at how many chips they have, it’s roughly on par. Now, I do think it’s concerning because up until recently, there were only three, four, maybe five companies that were part of this curve that could produce frontier models and they were all in the U.S. DeepSeek is the first time, the thing that really is notable, is the first time a company in China has been able to go toe-to-toe and produce the same kind of engineering innovations as companies like Anthropic or OpenAI or Google. That is actually very significant and that actually worries me.

Export Controls and National Security

MICHAEL FROMAN: Now, some argue that the emergence of DeepSeek means that export controls don’t work, can’t work, we should stop trying to control the export of our most advanced chips. Others say it means we should double down on export controls. Where do you stand on that?

DARIO AMODEI: So, I think it’s an implication of the framework I just gave that the export controls are actually quite essential because yes, there’s this cost reduction curve, but at every point along the curve, no matter how much the curve is shifted, it is always the case that the more chips you spend, the more money you spend, the better model you get. Right if it’s like, you know, okay, before I could spend a billion dollars and get a model that was okay, now I can spend a billion dollars and get a model that’s much better and I can get an okay model for $10 million, that doesn’t mean the export controls failed. That means stopping your adversaries from getting a billion dollar model just became a higher stakes thing because you can get a smarter model for a billion dollars.

And yes, DeepSeek was, you know, they had relatively small, you know, relatively small amount of compute consisting of chips that went around the export controls, some chips that were smuggled. But I think we’re heading for a world where we, OpenAI Google, are building billions, maybe tens of millions of chips costing tens of billions of dollars or more. It’s very hard for that to be smuggled.

If we put in place export controls, we actually may be able to stop that from happening in China. Whereas, if we don’t, I think they may be at parity with us. And so, you know, I was a big supporter of the diffusion rule.

I’ve been a big supporter of export controls for several years, even before DeepSeek came out because we saw this dynamic coming. And so, I think it’s actually one of the most essential things, not just AI, across all fields for the United States national security for us to prevent China from getting millions of these very powerful chips.

MICHAEL FROMAN: The diffusion rule, as I understand it, divided the world. This is a Biden administration, you know, that divided the world into three camps as to who could get access to what in terms of chips from us. Some worry that the countries that are not in the top tier are just going to be served by China and that China is going to end up running the AI infrastructure for the vast majority of the world.

DARIO AMODEI: So, my understanding of the diffusion rule and, you know, my understanding is the new administration is looking at it, but there are many parts that they’re sympathetic to. The way it actually sets things up is these tier two countries. So, tier one countries are like the majority of the developed world.

MICHAEL FROMAN: But not all.

DARIO AMODEI: Not all. Tier three is, you know, restricted countries like China or Russia.

Tier two are, you know, countries in the middle. Actually, you can have a very large number of chips in those countries if the companies hosting them are able to provide security affidavits and guarantees which basically say we are not a front company for China. We are not, you know, shipping the compute or what is done with the compute to China.

And so, there really is an opportunity to build a lot of U.S. chips, a lot of U.S. infrastructure in these countries as long as they comply with the security restrictions. I think the second piece of it is, yes, in theory companies could switch to using Chinese chips. But Chinese chips are actually quite inferior.

NVIDIA is way ahead of Huawei which is the main producer of chips for China. Like something like four years ahead. I think that gap is going to close eventually.

Over the course of, I don’t know, 10 or 20 years, probably the export controls may even have the impact of simulating China. But the tech stack is so deep. And I think the next 10 years during which we will stay strongly ahead in hardware are actually the critical period for establishing dominance in this technology which I would argue whoever establishes dominance in this technology will have military and economic dominance everywhere.

US-China AI Dialogue

MICHAEL FROMAN: The last administration launched a dialogue with China about AI. What are the prospects for such a dialogue? Where could we possibly agree with China? And do they care about responsible scaling?

DARIO AMODEI: So, you know, I would describe myself, and of course I wasn’t part of any of these conversations, but I heard a little about them.

I would describe myself as supportive of this dialogue but not especially optimistic that it will work. So, you know, the technology has so much economic and military potential that, you know, between companies in the U.S. or our democratic allies, you can imagine passing laws that create some restraint. When it’s just like two sides are racing to build this technology that has so much economic and military value, perhaps more than everything else put together, it’s hard to imagine them slowing down significantly.

I do think there are a few things. One is this risk of the AI models autonomously acting in ways that are not in line with human interests. Right? If you have a country of geniuses in a data center, a natural question, how could you not ask this question? Well, what is their intent? What are they planning to do? You would certainly ask, well, is someone controlling them? Are they acting on someone’s behalf? But you would also ask, well, what is their intent? And because we grow these systems, we don’t train them, I don’t think it’s safe to assume they’ll do exactly what their human designers or users want them to do.

So I think there’s real risk of that. I think it could be a threat to, you know, to kind of all of humanity and like issues of nuclear safety or nuclear proliferation, there’s probably some opportunity to take limited measures to help address that risk. So I’m relatively optimistic that maybe something narrow could be done.

Navigating U.S.-China Relations

The stronger the evidence is of that coming, you know, right now that’s a kind of speculative thing, but if strong evidence came that this was imminent, then maybe more collaboration with China would be possible. So I’m hopeful that we can try and do something in this space, but I don’t think we’re going to change the dynamic of national competition between the two.

Proposed Action Plan for the New Administration

MICHAEL FROMAN: You recently presented to, I guess, OSTP an action plan, a proposed action plan for the new administration, what they should do in this area. What are the main elements of that plan?

DARIO AMODEI: I think there’s three elements around kind of security and national security and three elements around opportunity.

So the first one is what we’ve been talking about, like making sure we keep these export controls in place. I honestly believe this is across all areas, not just AI, the most important policy for the national security of the United States.

The second thing is actually related to the responsible scaling plans, which is the U.S. government through the AISI has been basically testing models for national security risks such as biological and nuclear risks. The institute is probably misnamed. You call it safety institute. It makes it sound like trust and safety, but it’s really about measuring national security risks and we don’t have an opinion of exactly where that’s done or what it’s called, but I think some function that does that measurement seems very important. It’s also important even for measuring the capabilities of our adversaries, like they can also measure deep seeks models to see what dangers they might present, particularly if those models are used in the U.S., like what are they capable of? What might they do that’s dangerous?

Number three on the risk side is something we haven’t talked about, which is I am concerned about industrial espionage of the companies in the U.S., companies like Anthropic. China is known for large scale industrial espionage. We’re doing various things. There are things in our responsible scaling plan about better and better security measures, but many of these algorithmic secrets, there are $100 million secrets that are a few lines of code and I’m sure that there are folks trying to steal them and they may be succeeding. And so more help from the U.S. government in helping to defend our companies against this risk is very important.

On the opportunity side, I think the main three there are, one is the potential for the technology in the application layer, in things like healthcare. I think we have an extraordinary opportunity, as I said, to cure major diseases, major complex diseases that have been with us for hundreds or thousands of years and that we haven’t been able to do anything about yet. I think that will happen one way or another, but regulatory policy really could affect, you know, does it take five years for AI to help us produce all those cures and distribute to the world or does it take 30 years? And that’s a big difference for people who suffer from those diseases. So our view here is that the policies of today around healthcare, around FDA approval of drugs may not be appropriate for the fast progress we’re going to see and we may want to clear away some blockers.

The second is energy provision. If we’re going to stay ahead of China in this technology and other authoritarian adversaries, we need to build data centers and it’s better if we build those data centers in the U.S. or its allies than if we build them in countries that have divided loyalties where they could literally just abscond with a data center and say, oh, sorry, we’re on China’s side now. And so, some of this was done during the latter days of the Biden admin and I think it’s a bipartisan thing. I think that the Trump admin, this is one area of agreement. There’s interest in provisioning a lot more energy. We probably need across the industry maybe 50 gigawatts of additional energy by 2027 to fully power AI that has all the properties we’ve been talking about. 50 gigawatts, for those who don’t know, is about how much energy was added in aggregate to the U.S. grid in 2024.

And then the final thing is the economic side of things. As we talked about, I think the worries on the economic side are just as existential as the worries on the national security side. In the short run, we’re going to need to manage the disruption, even as the pie gets much larger. In the long run, as I’ve said, we’re going to need to think about a world where AI, and I don’t want to lie about this, where I really think where this is going is that AI is going to be better than almost all humans at almost all things.

We have to reckon with that world as soon as possible. For now, I think we just need to, the best thing we can do is measure to understand what’s going on. We released this thing called the Anthropic Economic Index that, in a privacy-preserving way, looks through and summarizes our usage to understand in what fields are people using it? Is it augmentative? Is it replacing? But in the long run, we’re really going to have, this is going to implicate questions about tax policy and distribution of wealth, right?

There’s this kind of alluring world where if the pie grows enough, there could be the resources to do a lot about this. Like let’s say, and this will sound crazy to this audience, but let’s say AI causes the economic growth rate to be 10% a year. Then suddenly the tax base is growing so much that, like, you can erase the deficit and, you know, maybe have all this left over to manage the probably enormous disruption that comes from the technology. So that will sound like crazy town, but, I just invite you to consider the hypothetical and start considering the possibility of crazy things like that now.

MICHAEL FROMAN: Crazy town. You heard it here first. Okay, let’s open it up to questions.

Q&A Session

AUDIENCE QUESTION: Thanks, Dario. This has been a really fascinating conversation.

DARIO AMODEI: Should I stand?

MICHAEL FROMAN: You just stand.

DARIO AMODEI: Okay, I will stand and get my steps in.

MICHAEL FROMAN: Just say who you are.

AUDIENCE QUESTION: I’m Adam Bancadeco. I enjoyed reading Machines, your essay last year, and then hearing you on Hard Fork on The Times, but also hearing this. The question I have for you is you sort of outlined the political and economic implications. But I’m curious to get a sense of how you’ve thought about the social and moral considerations that are going to effectively come, especially because I think most of the general public sees some of the chatbots and says, oh, it’s an improved Google search, but doesn’t really think about the downstream effects of the disruption in the labor market. I’m curious to get a sense of how do you sort of think about that intention with building a company, trying to build a commercial product?

DARIO AMODEI: First of all, I think this stuff is super important. Perhaps the most disturbing thing to me right now is the lack of awareness of the scope of what the technology is likely to break. I mean, I could just be wrong. I’m saying a bunch of crazy stuff, like, the answer could just be the general public is right and I’m wrong. I’m high on my own supply. I acknowledge that as possible. But let’s say it’s not the case.

What I’m seeing is there are these concentric circles of people realizing how big the technology could be. There’s probably maybe a few million people, very concentrated in Silicon Valley, but a few people high in the policy world who also hold these beliefs. Again, we don’t know yet whether we, they are right or wrong. But if we are right, the whole population, again, thinks of this stuff as chatbots. If we say this is dangerous, if we say this could replace all human work, it sounds crazy because what they’re looking at is something that, in some cases, seems pretty frivolous. But they don’t know what’s about to hit them. And so I think that actually keeps me up at night a lot and is why I’m kind of trying to spread the message to more people.

So I think awareness is step one. I think these questions around human labor and human work in a world where it is technologically possible to replicate the effects of the human mind, I think these are very deep questions. I don’t feel like I have the answer to them. I feel like, as you said, these are kind of moral questions, almost like questions about purpose. You can even say spiritual questions, right? And so we are all going to have to answer these questions together.

I’ll give you kind of the embryo of an answer I have, which is that somehow the idea of human’s self-worth, the tying of that to the ability to create economic value, there are aspects of that that are deeply embedded in our psychology, but there are aspects of that that are cultural. There’s a lot of things about that that work well. It’s created a modern precipitatory economy, but technology, as it often does, may kind of lay bare that illusion. It may be another moment like the moment we realize that the earth rotates around the sun instead of the sun rotating around the earth. Or there were many, many solar systems or organic material is not made up of different molecules than inorganic material.

So we just may have one of those moments and there may be a reckoning. And again, my answer is I am struck by how meaningful activities can be even when they are not generating economic value. I am struck by how much I can enjoy things that I am not the best in the world at. If the requirement is you have to be the best in the world at something in order for it to be somehow spiritually meaningful for you, I feel like you’ve taken a wrong turn. Like, I feel like there’s something wrong embedded in that assumption. And I say that as someone who spends a lot of time trying to be the best in the world at something I think is really important. But somehow our source of meaning is going to have to be something other than that.

AUDIENCE QUESTION: Cam Carey at the Brookings Institution. One of the things that leapt out at me from the UK AI safety report is the possibility that come 2030 or so that scaling up may run out of data. How do you then scale? How do you make the models smarter? And what are the limitations of that data? I mean, there’s a tremendous amount of text, video information that’s digitized. A tremendous amount that resides in our minds and in the universe that is not.

DARIO AMODEI: A couple answers on this. One is that in the last six months, there have been some innovations, actually not developed by us, first came from OpenAI actually. But others that we have made that obviate the need for as much data as we need before. These are the so-called reasoning models where they basically, they have thoughts. They start to think through the answers to complex questions. And then they train on kind of their own thoughts.

You can think about how humans do this, where sometimes I can learn things by, I’ll make a plan in my head. And then I’ll think about it again. And I’ll say, oh, actually, you know, on second thought, that doesn’t really make much sense. Like, what are you thinking, right? And then you kind of learn something from this. Of course, you also have to act in the world. You also have to act in the real world. But AIs have not been making use of that kind of cognition at all until recently. So far, that’s mostly applied to tasks like math and computer programming. But my view, without being too specific, is that it’s not going to be terribly difficult to extend that kind of thinking to a much wider range of tasks.

The second point is, even if we do run out of data in 2030, if the exponential continues for even two or three more years, it may get us to a point where we’re kind of already at the genius level. And that may be enough for a lot of these changes. And we may also be able to ask the models, hey, we have this problem. Human scientists weren’t able to solve it. Can you help us solve this problem?

I do still give a small likelihood that, for whatever reason, both of those things won’t work out or aren’t as they appear. And data could be one of the plausible things that could block us. I thought it was a very plausible blocker one or two years ago. I thought one or two years ago, if something would stop the show, this was in the top three of the list of things that would. But I think my potential skepticism here has been not completely refuted, but I think reasonably well refuted.

AUDIENCE QUESTION: What are the top three things that could stop the show?

DARIO AMODEI: At this point, I think the number one thing that could stop it would be an interruption to the supply of GPUs. If, for instance, the small disputed territory where all the GPUs produced had some military conflict, that would certainly do it. I think another thing would be if there’s a large enough disruption to the stock market that messes with the capitalization of these companies. Basically, a kind of belief that the technology will not move forward.

The Future of AI and Its Implications

And that kind of creates a self-fulfilling prophecy where there’s not enough capitalization. And third, I would say if I or we in the field are kind of wrong about the promisingness of this new paradigm of kind of learning from your own data. If somehow it’s not as broad as it seems or just there’s more to getting it right that we think there are some insights missing.

MICHAEL FROMAN: We’ll go to an online question. We’ll take the next question from Esther Dyson. I recognize that name.

Ms. Dyson, please unmute your line. Thank you. Apologies.

ESTHER DYSON: Esther Dyson, writing a book called Term Limits on term limits for people and for AIs and so forth. I have a question about this whole existential risk thing. It seems to me that the bigger risks honestly are humans who are even more unexplainable than AIs.

But humans and their business models using AIs. And then specifically, there’s the famous paperclip problem where you ask the AI to make paperclips and it does that to the exclusion of anything else. And this is slightly metaphorical, but the world seems to be going mad for data centers.

And it really is kind of draining resources from everything else to fund data centers, AI, data pools, whatever. And so in a sense, AI is creating a fitness function for society that is, I think, harming the value of humans, which is not just their intellectual capacity. That’s the end of the question.

DARIO AMODEI: Thank you. So I would say just as there are many different benefits of AI and every time we produce a new AI model, it has a long list of 10 benefits that we anticipated and then a bunch more that we didn’t. Every time we release a new model, there’s new use cases and customers were like, I didn’t even think of doing that with an AI system.

It is unfortunately also the case that we shouldn’t say this risk is a distraction from that risk. It just unfortunately is the case that there are many different risks to the AI systems. And if we want to get through this, we somehow have to deal with them all.

So I think it is a big risk that humans will misuse the AI systems. I think it is a big risk that the AI systems themselves, we may have difficulty controlling them. Again, to use the analogy of a country of geniuses and data center, we plop down a country of 10 million geniuses in some Antarctica or something.

We’re going to have multiple questions about what that will do to humanity. We’re going to ask, well, does some existing country own them? Is it doing their bidding and what will that do? Is the outcome of that beneficial? We’ll say, are there individuals who could misuse it? And we’ll say, what are the intentions of that country of geniuses itself? And then to get at the question you asked near the end, are there more distributed societal things? I certainly believe that if more and more of the world is, more and more of our energy is devoted to AI systems, it’ll be great. They’ll do things really efficiently, but also could that make some of our existing environmental problems worse? I think that’s a real risk.

And then you can say, well, will the AIs be better at helping us to solve our environmental problems? So we spend a bunch of energy and then the AI systems, it kind of, it turns, we end up better than we started if we’re able to solve it. So I’m optimistic that that will be the case, but that’s like another risk, like a number of things have to be true for it to turn out that way. So I just think we’re at a time of great change and therefore we have to make extraordinarily wise choices to get through it.

I recognize the name of the person asking the question, and I might get this wrong, but I think it was your father who said, I listened to a video of him, because I was a physicist and I listened to a video of him where he said, we have all these problems today and it seems like we can’t solve them. But I remember in my day, it really seemed like we had all these severe, severe problem, thinking of just World War II or the Cold War, nuclear annihilation, and somehow we made it through. So it doesn’t mean we will again, but… Yes, the woman in the back, there we are.

AI Welfare and Ethical Considerations

CARMEN DOMINGUEZ: Hi, my name is Carmen Dominguez. I’m an AI specialist with a background in development implementation and more recently focusing a bit more on the policy side. I hear you loudly clear on the lack of awareness generally of what is AI and what is not AI and what it can and cannot do.

But I’m going to skip over that. I do some science communication around that too. But my question today is around a few months ago, you brought on Kyle Fish as AI welfare researcher to look at sentience or lack thereof of future AI models and whether they might deserve more consideration and protections in the future.

If you could talk a bit about that, the reasoning for that, and if you have an equivalent human welfare research, to you going.

DARIO AMODEI: Yeah, so this is another one of those topics that’s going to make me sound completely insane. So it is actually my view that if we build these systems and they differ in many details from the way the human brain is built, but the count of neurons, the count of connections is strikingly similar.

Some of the concepts are strikingly similar. I have a functionalist view of moral welfare of the nature of experience, perhaps even of consciousness. And so I think we should at least consider the question of if we are building these systems and they do all kinds of things like humans as well as humans and seem to have a lot of the same cognitive capacities, if it quacks like a duck and it walks like a duck, maybe it’s a duck.

And we should really think about, do these things have real experience that’s meaningful in some way? If we’re deploying millions of them and we’re not thinking about the experience that they have, and they may not have any, it is a very hard question to answer. It’s something we should think about very seriously. And this isn’t just a philosophical question.

I was surprised to learn there are surprisingly practical things you can do. So something we’re thinking about starting to deploy is when we deploy our models in their deployment environments, just giving the model a button that says, I quit this job that the model can press, right? It’s just some kind of very basic preference framework where you say, if hypothesizing the model did have experience and that it hated the job enough, giving it the ability to press the button, I quit this job. If you find the models pressing this button a lot for things that are really unpleasant, maybe you should pay some, it doesn’t mean you’re convinced, but maybe you should pay some attention to it.

It sounds crazy, I know. It’s probably the craziest thing I’ve said so far. Way in the back there.

AI in Social Sciences and Public Health

TROOPER SANDERS: Trooper, yeah. Hi, Trooper Sanders. You talked about the excitement of AI and medical science, biology, chemistry, et cetera.

I was wondering if you could say, what is there any excitement around the social sciences? So most of healthcare is done outside of the pillbox and the exam room. Public health involves a number of other areas. Can you say anything about that side of things?

DARIO AMODEI: Yeah, I mean, if I think about epidemiology, when I was in grad school, there was a project being done by the Gates Foundation to use mathematical and computational methods around epidemiology.

I think they were planning to use it to help eradicate malaria, polio, other areas. The quantity of data that we get and the ability to pull all the pieces together and understand what’s going on in epidemic, I bet that could benefit hugely from AI. The clinical trial process.

We’ve already seen things like this. So actually, this is something Anthropic has done with Novo Nordisk, the maker of Ozempic and other drugs. At the end of a clinical trial, you have to write a clinical study report.

It summarizes adverse incidents, does all the statistical analysis to present to the FDA or other regulatory agencies for whether to approve the drug. Typically, this takes about 10 weeks. They’ve started using our model for this and the model takes about 10 minutes to write the clinical study report and humans take about three days to check it.

And the quality, at least as we’ve seen in early studies, that doesn’t determine everything, has been deemed to be comparable to what humans are able to do with the 10 week process. So we need new clinical trials. There’s a lot of social science problems around that.

There’s a lot of regulatory problems. I write about that in a bit in yesterday that I think those things are going to be the thing that limit the rate of progress. But even within things like clinical trials, I think the AI systems will be able to help a lot in if not dissolving those questions, at least radically simplifying them.

Illicit Trade and National Security Concerns

LOUISE SHELLEY: Yes, right here. Here’s a microphone coming. I’m Louise Shelley.

I’m an expert on illicit trade from George Mason University. Next week, there is a global summit of the OECD on illicit trade. But what you’ve talked about is not what I expected to hear on this problem of smuggling of parts and it’s on no one’s radar screen.

What happens when you’re talking about it because it’s not reaching the community that is needing to protect this illicit trade?

DARIO AMODEI: I didn’t hear the last part of the question.

LOUISE SHELLEY: So how come these issues aren’t on the agenda of those people concerned about illicit trade?

DARIO AMODEI: Yeah, I think my answer to that is it should be on the radar of those people. Again, I have a worldview here that not everyone shares and I may be right or I may be wrong.

But all I can say is if this worldview is correct, then we should be worrying a lot more about smuggling these GPUs than we’re worried about smuggling guns or even drones or fentanyl or whatever. But if you were to smuggle five million of these to China and to be clear, that’s like $20 billion of value or something like that. That would drastically change the national security balance of the world.

I think it’s the most important thing. So again, this is a dilemma of am I just crazy or does the world have a big, big awareness problem here? And if the world has a big, big awareness problem here, then a downstream consequence of that is we’re focusing on all these other things. And because when you say illicit trade, there’s certain things that people have been focusing on for a long time.

This is a new thing, but that doesn’t mean it’s not the most important thing.

AI and National Security Collaboration

ALAN RALL: Thank you. Alan Rall, practicing lawyer, lecturer at Harvard Law School and future useless person.

So are we all. So I’d like to follow up on your various comments on national security. You mentioned the Artificial Intelligence Security Institute and its testing.

The Biden executive order on AI had mandatory reporting of acquisition or development of super capable 10 to the 26th flop dual use foundation models. But my question is, how do you engage? How does Anthropic, the AI community, developers of these super capable models, how do they engage with, let’s just say, the US national security community, the intelligence community, and practically, what does that mean for the development of AI? And if you tell me you’d have to kill me, I don’t need to know that badly.

DARIO AMODEI: Yeah, so I think there are a few things here. One is typically, Anthropic in particular, although the other companies have started doing similar things. Whenever we develop a new model, we have a team within Anthropic called the Frontier Red Team. Some of this is happening, you know, testing with the AI safety and security institutes.

But, you know, we work in collaboration. We develop some stuff. They develop some stuff.

But the general flow has been when we test the models for things like biological risk, or cyber risk, or, you know, chemical or radiological risk, we’ll typically go to people in the national security community and say, hey, this is where the models are at in terms of these particular capabilities. You know, you guys should know about this because, you know, you’re the ones who are responsible for detecting the bad actors who would do this with the models. You know what they’re capable of now.

You might therefore have a sense of what the models can do that is additive or augmentative to their current capabilities, right? The part of it we missed is like, you know, we’re not counterterrorism experts. We’re not experts on all the bad guys in the world and what their capabilities are and what, you know, large language models would add to the picture. And so we’ve had a very productive dialogue with them on these issues.

The other topic on which we’ve talked to them about is the security of the companies themselves. You know, this was one of the things in our kind of OSTP submission of, you know, making this more formal, making this something that the US government does as a matter of course. But, you know, if we’re worried that we’re going to be attacked digitally or, you know, or, you know, via kind of, you know, human means, insider threat, then, you know, we’ll often talk to the national security community about that.

And I think the, you know, the third kind of interaction is, you know, about the national security implications of the models, right? We’ve been, you know, these things that I’m saying publicly now, I’ve been, you know, I’ve been saying them in some form to some people for quite a while. Then I think the fourth thing is there’s an opportunity to apply the models to enhance our national security. This is something that I and Anthropic have been supportive of, although we want to make sure that there’s the right guardrails, right? On one hand, I think, you know, if we don’t apply these technologies for national security, we are going to be defenseless against our adversaries.

Limits on AI in National Security

On the other hand, I think everyone believes that there should be limits. You know, I don’t think there’s anyone who thinks we should hook up AI systems to nuclear weapons and let them fire nuclear weapons without humans being in the loop. That’s the plot of Dr. Strangelove.

“Yeah, that is literally the plot of Dr. Strangelove. So somewhere between there, there is some ground. And we’re kind of still working on defining that.”

It’s one of the things where we hope to kind of be leaders in defining what the appropriate use of AI for national security is. But that’s another area where we’ve had interaction with the national security community. Your comment a couple of minutes ago about trying to understand the experience of the AI models has been sinking in for me.

What It Means to Be Human

MICHAEL FROMAN: So let me just conclude with one final question, which is in the world that you envisage, what does it mean to be human?

DARIO AMODEI: I think my picture of it, the thing that seems most human to me is, you know, struggling through our relationships with other humans, our obligations to them, how we have to treat them, the difficulties we have in our relationships with other humans and how we overcome those difficulties. When I think of both things that people are proud of doing and the biggest mistakes people have made, they almost always relate to that.

And AI systems maybe can help us to do that better. But I think that will always be one of the quintessential challenges of being human.

And I think maybe the second challenge is the ambition to do very difficult things, which again, I will repeat, I think will ultimately be unaffected by the existence of AI systems that are smarter than us and can do things that we cannot do.

I again think of like, human chess champions are still celebrities. I can learn to swim or learn to play tennis and the fact that I am not the world champion does not negate the meaning of those activities. And even things that I might do over 50 years, over 100 years, I want those things to retain their meaning and the ability of humans to strive towards these things, not to give up. I think those two things are maybe what I would identify.

MICHAEL FROMAN: Please join me in thanking Dario Amodei.

Related Posts

• Mo Gawdat: How to Stay Human in the Age of AI @ Dragonfly Summit (Transcript)
• Transcript: The ONLY Trait For Success In The AI Era – Aravind Srinivas on Silicon Valley Girl Podcast
• Transcript: Mark Zuckerberg on AI Glasses, Superintelligence, Neural Control, and More
• Lessons from Apple ID Hacks: What Transcription & Media Sites Should Do to Secure Their Users’ Accounts
• Transcript: Demis Hassabis on AI, Creativity, and a Golden Age of Science – All-In Summit