Transcript: The Next 50 Years: Humanity, AI, Power w/ Yuval Noah Harari

And what we see in history is that we do not know, at least I don’t know, of a single case of a large-scale democracy before the modern era. We have plenty of examples of small-scale democracies in the ancient and medieval worlds. We have city-states like ancient Athens or medieval Florence. We have lots of tribes and towns and villages run democratically. It seems that at least in the Stone Age, democracy was the most common system of government. There were few tribes with a single dictator controlling everybody.

But as human systems grew larger and you had kingdoms with millions of people spread over thousands of kilometers, it was impossible to hold a conversation anymore. You could do it in a small city like Athens. Everybody can gather together in the main square and discuss. But how do you do it if even in a relatively small kingdom like, say, Portugal in the Middle Ages? Can’t be done. And therefore we don’t have any example of a large-scale democracy before the rise of modern information technology — first with print newspapers, then radio and television and the internet.

And these technologies are not kind of a silver bullet, a side dish, like you have the democratic banquet, and yes, we also have these newspapers and radio. They are the basis. They provide the means for millions of people to hold a meaningful conversation in real time about whether to go to war or make peace. And every time there is a major change in these information technologies, you have an earthquake in the building that is built on top of that, which is democracy. And this is what we have been seeing over the last 10 years, for instance, as social media became maybe the most important information technology, causing a massive earthquake in democracies all over the world as a result.

AI as Agent, Not Tool

TOM PARKER: Yeah, one thing you said there was the choice between going to war or making peace. And actually, I want to get into the definition of AI because a lot of people call it a tool, but you argue that it’s absolutely an agent.

YUVAL NOAH HARARI: Yes.

TOM PARKER: And in fact, you say it’s a knife that can decide by itself whether to cut salad or commit murder. And I have to say, it would be a bizarre cooking show if that was the case, where knives had that ability to cut salads or commit murders. So it is a dangerous thought. Can we expand on that? Are we choosing the right narrative at the moment with AI by saying it’s a tool that humans can still use and that humans are in the loop? Or do we just need to say it’s an agent and they can’t talk?

YUVAL NOAH HARARI: It’s an agent. I mean, if it’s not an agent, then it’s not an AI, and all the hundreds of billions of dollars that are being poured into its development will be wasted. Because the expectations, what people expect this technology to do, the reasons they invest hundreds of billions of dollars is because they think it will be an agent, not a tool.

You listen to the narrative coming out of places like Silicon Valley and there is an inherent contradiction in the narrative. They tell us we will create a god and it will be our slave. And this doesn’t make sense. If it is a god, it cannot remain a slave. And if it is a slave, it means it has no godlike abilities. The assumption of most of the leading companies and most of the leading experts also outside the companies is that they’re correct about the first half of the sentence. They are creating a god. A god in the sense of something with superhuman abilities. And this is essentially an agent.

The difference between tool and agent: a tool is something that cannot make decisions by itself. We have to decide what to do with it, and it cannot invent new ideas by itself. The printing press was a tool. The printing press by itself — Gutenberg brings print technology to Europe in the 15th century — the printing press cannot decide by itself, “I want to print the Bible, and tomorrow I’ll print the Quran, and next week I’ll print Confucius.” No, Gutenberg decides “I’ll print the Bible,” and the printing press is just a tool to realize the decision of Gutenberg. Similarly, the printing press cannot invent a new idea. The printing press cannot invent the radio. It cannot invent a book. You need humans to write the book and then the printing press copies it.

Similarly, an atom bomb is a tool. An atom bomb in the 1940s could not decide by itself whether to bomb Hiroshima or to bomb Tokyo. You needed a human to decide, “Let’s bomb Hiroshima.” And the atom bomb could not invent by itself the hydrogen bomb — impossible.

An agent, in contrast, is something that can make decisions by itself and invent new ideas by itself. AI is an agent. An AI weapon can decide what to bomb and it can invent the next weapon. Similarly, there is a lot of interest now in creating AI agents in the financial and economic sphere, creating non-human corporations. Imagine a corporation that manages money and employees and has no humans as executives or shareholders or trustees, only AIs. This is a new type of agent that never existed before in history.

Now, is it good? Is it bad? Leave it aside. The first thing to grasp is that this is unprecedented. We had other agents around us like horses and cows and chickens and birds, lots of animal agents that can make their decisions, but we never encountered an agent that understands our language and that is better than us at things like finance or law or religion.

Again, we used language over thousands of years to create the control systems of the world. The control systems of the world are these huge bureaucratic networks ultimately based on language — like banks, like the stock market, like the Catholic Church, like the legal system. We never really liked these bureaucratic systems. Humans generally don’t like bureaucracy very much, but we can’t live without them. They are the basis for our power. AI is a bureaucratic native.

What is likely to happen — what is already happening — don’t expect to see killer robots running in the streets shooting people. This is not how AI will take over the world. It will be the AI bureaucrats. We control the world because of these networks of bureaucracy and we are not very good at it. We only control them because there was nobody else on the planet that could take them from us. We are bad at finance, but the horses are even worse. We are bad at managing the legal system, but the chickens are even worse. Now, if somebody comes along that is better than us at bureaucracy, at finance, at law — that’s the question. What will happen to us at that moment?

Legal Personhood and the Decisions We Cannot Take Back

TOM PARKER: Well, let’s look at the legal side of it a bit and the fact that AI as an agent currently doesn’t have any legal personhood. We don’t give it the same human legalities, do we? But you’ve said before that if we don’t decide where AI sits from a legal standpoint — when we’re talking about autonomous vehicles, etc. — someone else will make that decision for us in 10 years, and we’re out of that power loop. That’s a pretty scary thing.

But I want to pull back a bit here because widening that out, what is the one decision on the table right now that you believe, once made, humanity will not be able to take back? Is it that legal personhood of AI, or is there something else?

Legal Personhood for AI: The Fox Enters the Chicken Coop

YUVAL NOAH HARARI: Yes, I would say that there is a lot of attention on the technological race to develop the technical side of AI, to make it faster, more efficient, more powerful. But I would focus also on the legal and political aspect, and especially on the question whether we grant legal personhood to AIs.

Now, what is legal personhood? A person in the legal system is someone who, for instance, can open a bank account, someone who can be a player, an independent player in the financial system, make investments, someone who can be a player in the legal system, for instance, sue you in court.

Now, until today, there are just 2 kinds of legal persons in the world. You have human beings, of course, we are persons. I can open a bank account, I can sue you in court, I can donate money to a politician, I’m a legal person. But most legal systems in the world also recognized in the modern era another kind of legal person which are corporations. According to not all but most legal systems, corporations like Google or Facebook or Toyota or Mercedes-Benz, they are legal persons. The corporation can open a bank account. The corporation can sue you in court. The corporation can make a political donation.

Now until today, this was legal fiction because all the decisions of the corporation were actually made by some human being. If Google decides to acquire another company, who actually made the decision? Not Google, but the human executives, the human shareholders, the human engineers made a decision. If Google decides to sue you in court, it was some human employed by Google. There was no real Google. Now there can be.

If we grant AIs legal personhood — and just a month ago, Argentina, the government of Argentina announced that it is going to grant legal personhood to AIs — so they can start having operating non-human corporations. Corporations again that can employ people, can sue you in court, can do anything a corporation does and has no humans.

Now, 10 years ago, this would just be a crazy idea because if there are no humans, who makes the decisions? You can pass a law in parliament that you can have a non-human corporation, but it can’t function. Now it’s technically possible. AI can manage, for instance, a bank account and decide to invest here and buy this and buy that.

Once we do that, that’s the moment when the fox enters the chicken coop. Again, the basic realization should be most humans are not very good in understanding finance and law and all these complex bureaucratic systems. No human is able to remember all the laws of Portugal. No human is able to track all the financial transactions in the market that happened today. And AI can do that. AIs are bureaucratic natives. If we give them the legal power, then they will likely take these systems over.

And if we don’t make any decisions, it will just happen by itself as we already saw on social media. On social media, we never made a decision, “let’s grant AIs legal personhood on social media.” But social media is full of AI persons, of bots that impersonate people. When somebody sends you something online, you have no idea whether that somebody is a human being or an AI. Social media is the first system in the world where AIs actually function as persons even though we never decided that we want to allow that. It just happened. And the results are not very good.

The Editors Are Now Algorithms

Media also ultimately is a kind of bureaucratic system. In the 20th century, it was managed by human beings. The most important persons in the media landscape of the 20th century were the editors — the editors of the newspapers, of the radio shows, of the TV stations — because they decided what everybody will be thinking and talking about. They control the conversation. When you decide, out of the million things that happened today in the world, what will be the 10 things that will appear on the front page of tomorrow’s newspaper and what will be the main headline. And then tomorrow you have millions of people, they get the newspaper, they read it, and they start talking about it and thinking about it. This was immense power.

So human editors were extremely important persons in the 20th century. Lenin, for instance, before he was dictator of the Soviet Union, his one job was editor of the newspaper Iskra. Benito Mussolini started as a socialist, actually, journalist, then switched to being a far-right journalist, then became editor of the far-right newspaper Il Popolo d’Italia, and from that position, his next job was dictator of Italy. So it was like journalist, editor, dictator.

Now, if you ask yourself, who are the most important editors today in the world? What are their names? They have no names because they’re algorithms. Who manages, who decides what people will see on their social media feed, on X, on TikTok, on Instagram? It’s not human beings. The entities that control the bureaucracy of the media are already AIs.

TOM PARKER: Yeah, you’ve said that before, haven’t you? And we’ve been talking a lot today about the shifts in power, and you said that essentially whoever has controlled various different parts of our evolution and revolutions have controlled either energy, they’ve controlled manufacturing, and now it’s those that control data. So what does a 21st century arms race actually look like in practice if it’s those that control data that are at the top of that power mountain?

The AI Arms Race: A New Kind of Empire

YUVAL NOAH HARARI: What we are seeing now is that there is an arms race really between 2 countries, China and the US, and a very small number of companies. They control most of the world’s data. They lead the race to develop superintelligence, to develop better and better AI models. And this really gives them the power to control the world.

If you look at the last big revolution in technology in the 19th century, the Industrial Revolution, the few countries who led the Industrial Revolution, this gave them the power to conquer and exploit the rest of the world. There was just no match if somebody comes with the power of modern industry with steam guns and machine guns, and the other side has spears and horses, it’s no match. And in the 19th century, even very small countries like Belgium, once they industrialized, they had the ability to build a huge empire in Africa, what is today Congo. The thinking in the 19th century was that industry and empire go together.

Now with AI, it can be even far more extreme. We can have 2 countries that basically control all the infrastructure that everything runs on. From military to civilian technology, everything will run on AI. And there are 2 unique characteristics of AI that make this arms race different from every previous arms race or from every previous imperial competition.

In the past, even if you were a very strong empire, you could not concentrate all the power in the metropolis, in the imperial hub. Like if you’re the Roman Empire and you control the entire Mediterranean, the main economic asset in ancient times was land. The land where you grow wheat and olives and grapes. And you couldn’t technically take the wheat fields of Egypt and the olive groves of Iberia and move them to Italy. Impossible. So even at the height of the Roman Empire, a lot of power remained in the provinces. And ultimately it shifted to the provinces. So in the late Roman Empire, the city of Rome and Italy are abandoned and the center shifts to the eastern Mediterranean, which was the most fertile and important area in terms of land.

You move forward, say to the 19th century in the British Empire, with industry you can concentrate power a lot more easily. Because you have all the main industries concentrated in the British island, but still you cannot move everything to Britain. You cannot move the oil fields of Iraq to Yorkshire. You cannot move the rubber plantations of Malaya to Cornwall. It’s impossible. So you concentrate industrial production in Britain, but a lot of power still remains in the provinces.

With an AI empire, it’s different. You can technically concentrate all the world’s information and all the code that controls everything in just one country or 2 countries. So this makes the potential for imperial concentration of power much bigger than in any previous era.

The Kill Switch: AI’s Unprecedented Power Over Sovereignty

And the other thing is that you can create a system where everything runs on AI infrastructure all over the world, but there is a kill switch in the imperial hub. When a Roman merchant sold a steel sword to some Gothic tribe, the Romans lost control of the sword. The Goths could then use the swords to fight against the Romans and kill Roman legionnaires. There wasn’t a button in Rome that the emperor presses and all the steel swords we sold to these Goths and Vandals and Franks stop working. Can’t do that.

Similarly, in the 19th century, if British merchants sell rifles to Afghan tribesmen in the mountains of the Hindu Kush, and the Afghans then use these rifles to defend Afghanistan from a British invasion, there is no button in London that Queen Victoria presses and all the rifles in Afghanistan stop working.

That’s the case with AI. If it is managed in a certain way, say the Americans and Chinese can sell all the world AI weapons and AI civilian technology and everything runs on that — the governments, the industry, the militaries. But if you do something they don’t like, they just press a button and everything stops working.

And we were given some foretaste of that, for instance, in the war in Ukraine, with weapon systems that were based on Starlink. There were certain incidents where Elon Musk just pressed a button and the Ukrainian drones stopped working. Now it should be said that Starlink did an immense positive service to the Ukrainians, so I don’t want to come across as too critical of that. But just to illustrate the point that AI weapons, for instance, are not like machine guns. They are embedded within a whole network.

And that’s why there is so much — I heard that you this morning discussed the issue of sovereignty. Sovereignty in the age of AI is much more complicated than in the industrial age. Because the nature of the technology is such that at least in certain architectures, there is always a kill switch in the imperial hub. And your only choice is whether you are willing to become a subservient vassal of the empire or to be left completely outside the race as barbarians.

There is of course a choice of trying to build an alternative to the American and Chinese systems, but the last train is about to leave the station. If you want to do it, you have to do it now.

TOM PARKER: Well, sticking on a slight wartime footing, I want to actually look at truth in an AI world, because Winston Churchill once said that “the truth is so precious that it should always be protected by a bodyguard of lies.” And that was about how there was a lot of, I suppose, Axis powers that were looking at where Britain was going to invade. So they used to lie a lot about where Britain was going to invade to hide the truth.

Now, have these bodyguards of lies, when we’re looking at truth in an AI era, turned on the truth, turned their guns on the truth, and now we have no idea what truth is because there can be these deepfakes that are sent out around the world by media organizations, by politicians, by every single institution that we trusted? How difficult is it for our generation and future generations to even know what is true? Or does the truth even matter?

Truth, Fiction, and the Limits of Human Understanding

Yuval Noah Harari: The truth, of course, matters very much. It’s the basis for everything. The problem throughout history is that in the contest between truth and fiction and fantasy, fiction has inherent advantages. The truth is costly. If you want to know the truth about anything, you need to spend a lot of time and energy and effort looking for it and fact-checking and investigating. Fiction is very, very cheap. You just write anything you want.

The truth has another problem, which is that it tends to be complicated, and people don’t like complicated stories. Fiction can be made as simple as you would like it to be. Finally, the truth is often not just complicated, it’s also painful. There are many things people don’t really want to know about themselves, about their nation, about their group, whatever. Fiction can be made as flattering, as pleasant as you want it to be.

So in a competition between the truth, which is costly and complicated and sometimes painful, and fiction, which is cheap and simple and very attractive and flattering, fiction tends to win. Unless we make the special effort to support the truth. And over history, humans again and again made the effort to build institutions to search for the truth and to protect it. Journalism, for instance, is one such institution. The scientific establishment is another such institution.

There are huge advantages, of course, to knowing the truth. Everybody in the end wants to know at least some truths, at least about yourself. You want to know the truth about your life, otherwise you will never be happy. If you don’t know yourself, then for instance, you will never know what are the sources of misery and what are the sources of happiness in your life, and then even if you are the most powerful person in the world, if you don’t know the truth about yourself, you’ll be miserable. Because you will not know how to use all your power to make yourself happy.

So there is an inherent yearning in human beings to know the truth, but it is difficult. There are challenges. And some people fantasize AI will tell us the truth about the world. It will not. It will create a much more complicated world in which it will be much more difficult to know the truth.

From the Stone Age to AI: The Paradox of Progress

Yuval Noah Harari: And this is not something new. If we look at the trajectory of history from the Stone Age until today, what we see is that humans learn a lot of things about the world. We know all kinds of things about physics and biology that people in the Stone Age didn’t know. But our advances in knowledge also lead to creating more and more complicated systems that make it more difficult to understand our lives.

If you ask who understands their lives better, an average hunter-gatherer 50,000 years ago or the average person today, the average hunter-gatherer understood their lives much better. There are many things they didn’t know. They tried to understand what controls the migration patterns of the animals they hunt, what controls the growth cycle of plants, what causes disease, what causes old age. They didn’t know. There are many things they didn’t know about their lives. We know all these things. We know what controls the migration patterns of animals or causes most diseases.

But we understand our lives much worse because our lives are no longer dominated just by animals and plants. They are dominated by these huge systems we have created, like the financial system, like the political system. And very few people understand the financial system or the political system or the legal system that controls their lives.

And AI is likely to take this to extreme. It will reveal some truths that we don’t know, but it will create much more complicated systems of control that most humans will have almost zero chance of understanding. And this is one of the dangers of the AI revolution — that we will be reduced to, say, the position of the horses in the financial system.

The lives of horses today are controlled ultimately by the financial system, but the horses don’t even know that this system exists. They look around and say, “We see trees, we see dogs, we see cows, we see humans, we see houses. We don’t see any financial system.” It’s invisible to them. It’s too complex for the brain of a horse to understand what is the stock market and what are corporations and what are bonds and stocks and all these things that ultimately control the lives of the horses.

Now, most humans today also don’t understand the financial system. I think a generous estimate is that, I don’t know, 5% of humanity understand it. In 10 years, the number could be zero. AIs will just make the financial system so complicated mathematically and so fast. Think about bankers that never sleep. Think about investors that never go on vacation. They don’t have family. All their lives is just running the financial system. And we could be very, very soon, in our lifetimes, in a situation where this just becomes too complicated for the human brain to process. And there is not a single human being on Earth that still understands finance.

The Power of Words — and Its Limits

Tom Parker: I bet there are a few financiers in this room that are feeling a bit awkward at the moment. But at Davos, actually, you even mused that your entire life’s work — persuasion through words, that’s what you’ve done your entire life — might also be at its limit.

Yuval Noah Harari: Yes.

Tom Parker: Because AI can manipulate and create through words. So what power of persuasion does a modern society and a future society have if we lose something that you’ve dedicated your entire life to?

Yuval Noah Harari: I work with words. I write books. I tell stories. I work with words. And when I look at the development of AI over the last 10 years — and I think I’m still better at writing than AI, but just a bit — it’s already better, I think, than most humans. I wouldn’t be surprised if in 10 years it’s much better than me. That it will take over language and everything that is made of language.

And the really big question here is whether AI can think. And this is a deep philosophical question because it really asks, what is thinking? There are two options to understand what thinking is.

One option is that thinking simply means putting words in a particular order, putting language tokens in order — like “all humans are mortal, Socrates is a human, therefore Socrates is mortal.” This is a logical syllogism. It’s made by putting words in order. This is something that AI is already able to do far better than most of us. We can put 20 words in order. AI can put 20,000 words in order. It’s just so much better. If this is what thinking is all about, AI will take over even thinking.

And you have people say, “No, it’s just a glorified autocomplete. It just predicts the next word in a sentence.” But when I observe what’s happening inside my own mind, what I find there is some kind of LLM that just predicts the next word in the sentence I’m saying. This sentence that I just now said to you — when I started saying it, I did not know how it would end. Why did I say the sentence I’m saying and not some other words? I don’t know. In my mind, one word comes after another, and sometimes as a public speaker, I’m terrified that I don’t know if I’ll know what to say next. Maybe I have a block. Because really — and this is true of everybody, I think — some people think in images, not in words. But in that case also, where do the images come from?

You can test it afterwards for yourself. Just try to observe how you think. Where do the words come from? The next word — why this word? Why not some other word? We don’t really know. So saying that AI is just predicting the next word — okay, how is it different from us?

Can AI Feel? The Question of Consciousness

Yuval Noah Harari: Now there is an alternative view of thinking. That thinking is not about putting words in order, it’s about the feelings around these words. You can say the same sentence twice and feel completely different about it. And that the power of thinking comes from the stomach, not from the brain. It’s not about the words, it’s about the feelings that prompt the words and that are inspired or evoked by the words. And here the big question is, can AI feel?

We know that AI can manipulate language better than us, but does it feel anything? And we have no way — science has so far failed to come up with a good model for consciousness, for feeling, and a good test for that.

And this is going to be a major issue because we already see that AI is mastering not just finance and not just laws and warfare, but also relationships. More and more people develop intimate relationships with AIs, especially young people. Many of them now say, “My best friend in the world is an AI. I tell my AI friend things I don’t tell my parents, I don’t tell my older brothers, I don’t tell my teachers. I tell my AI friend.” And you already have people with AI boyfriends and girlfriends.

And the big question is, what is behind it? We know that the AIs are becoming very good at creating the impression that they feel. They use language to do that. An AI can today tell you, “I love you.” And if you ask the AI, “Explain to me how you feel when you’re in love. I don’t think that you really love me. I think you just say the words. So convince me. Describe to me how love really feels like.” For the AI, it’s going to be a very easy task because the AI has read maybe all the love poems ever written and remembers them, has read all the plays of Shakespeare and all the Hollywood comedies and all the psychology books. And it can describe love better, maybe in 5 or 10 years, than any human poet or psychologist. But is there anything behind the words? We don’t know.

Again, this goes back to the beginning of our conversation when I said AI might not be a machine that masters language. It might be language itself liberating itself from any material substrate. It doesn’t depend on a flesh and blood organism. It doesn’t depend on this machine or that machine. It’s just language developing and spreading and doing stuff in the universe.

Raising Future Generations in an AI World

Tom Parker: You mentioned there this scary experiment. We’ve only got a couple of minutes left, but the interaction with our future generations and AI that has created this emotive relationship, but essentially there aren’t emotions behind it. In 50 years’ time, how would you like historians to look back, talk about this particular time right now — the way you’ve spoken about cognitive revolutions — and say that we did something to re-educate our future generations on how to interact with humans, how to have empathy, so that we’re not lulled into relationships with AI? And very quickly, because we’ve only got a couple of minutes left, it’s a big question, I apologize.

The Wisdom Revolution

Yuval Noah Harari: I hope they will remember it as maybe something like the wisdom revolution. And this is true of — think of it in economic terms. Every time something becomes abundant and cheap, the question is, okay, what’s the next bottleneck? When energy becomes cheap, what’s the next bottleneck?

Now the AI revolution is about making intelligence very cheap and abundant. Previously, intelligence was rare and expensive. Intelligence is the ability to solve a problem. You have a goal — how do I get to that goal? You want to cure cancer, you need intelligence to cure cancer. You want to make a billion dollars, you need intelligence. You want to get to Mars, you need intelligence. Now in a world where intelligence has become very cheap and abundant, you can solve anything. What’s the bottleneck? The bottleneck becomes the question, “Okay, what do I want to solve? I can have anything I want — what do I want?”

And you have all these fairy tales — almost every human culture has these fairy tales of, say, a genie that pops out of some lamp and tells you, “You have three wishes, but think very carefully what you’re going to ask.” In almost all the fairy tales, it goes wrong. People ask for the wrong thing.

This is the difference between intelligence and wisdom. Intelligence is the genie. The genie will do anything, all your dreams will come true, but you need wisdom to choose the dreams. You need the wisdom to know what do I really need to solve.

And I hope that humanity will rise up to that challenge. Hopefully the AIs will not get out of our control. We’ll design it in the right way. We are not doing it right now. A good statistic to remember is — the companies that produce AI, that design these AIs, what percentage of their budget in terms of money and employees is invested in safety, making sure the AIs are safe, versus what is invested in making sure the AI is faster, more powerful, more sophisticated? At the present moment, it’s about 100 to 1. For every $100 million they spend on making the AI more powerful, they spend only a million making sure it’s safe. No other industry on earth works like that. We would not tolerate it in the energy sector, in pharmaceuticals, in the car industry. This is what is happening with AI.

Hopefully we’ll change that and we’ll produce safe intelligence, and also work on our own wisdom so that we know what goals to set and what desires to pursue.

Tom Parker: Well, that’s all we’ve got time for. Yuval Noah Harari, thank you for sharing your wisdom with all of us, and I’m sure everyone has appreciated this evening.

Yuval Noah Harari: Thank you.

Related Posts

• Transcript: Maya Angelou’s Greatest Life Lessons on Oprah Podcast
• How to Stand Out in an AI World by Thinking Better, Not Faster: Sol Rashidi (Transcript)
• Modern Wisdom: w/ Alex Petkas on True Story Of The Odyssey (Transcript)
• Reconstruction: Malcolm Gladwell Interviews Barack Obama (Transcript)
• Transcript: “How to Win Without Crushing Your Soul” – Graham Weaver