AI Debate on Modern Wisdom: w/ Zack Kass, Liv Boeree & Aric Floyd (Transcript)

The people who are very strongly at 90% use it as a cudgel; the people who think it’s silly say it’s just not that helpful. I think we can all agree — well, maybe you don’t — but I think we can all agree that the future is far from certain and it can be very precarious if we don’t get our act together.

ZACK KASS: Do I strike you as a fatalist?

LIV BOEREE: What do you mean by fatalist?

ZACK KASS: I mean, I certainly don’t think that the future is certain by any measure.

ARIC FLOYD: No.

ZACK KASS: Yeah, okay, but you were saying “I think we all agree.” You looked at me as though I might not agree.

LIV BOEREE: Well, I just don’t want to presume. We’re early in the conversation. I don’t want to make presumptions.

CHRIS WILLIAMSON: What do you think, Aric?

ARIC FLOYD: I definitely agree that anything above 1%, when we’re talking about an outcome as catastrophic as extinction or mass human disempowerment, is just way more than we should tolerate. And that’s a revealed preference of society. We spend a good amount of our GDP preventing nuclear catastrophe. We should probably be spending a lot more preventing the next pandemic. Certainly people were on board with that in 2021.

I think the weird thing about AI is that this thing has so much upside and downside. It’s very easy to just be anti-nuke and to be anti-viruses — there aren’t many people who are like “virus rights,” you know? But with AI, it’s the whole package bundled up together: the exact same thing that could free people to have an incredible Tuesday free from the obligation to work and just able to pursue joy, whatever that means to them, is the same thing that an oligarch in Russia could use to have an army of workers loyal to them, increasing their share of the world GDP without any other human needing to approve it.

CHRIS WILLIAMSON: Post-scarcity, totalitarianism, and destruction all just swimming around in the same pot.

ARIC FLOYD: Yeah.

ZACK KASS: Some potential for techno-pestilence, also.

LIV BOEREE: Right.

The Jagged Perimeter and Political Protection

ZACK KASS: I haven’t heard that. I like that one. Okay. But there are two problems here with the framing. One, AI is not one thing, it’s lots of things. And robotics obviously introduces all sorts of new modalities, but it has a jagged perimeter. So when people ask, “How is healthcare going to change by 2040?” — unfortunately, going into the hospital will probably still suck.

There is so much to unwind about healthcare in the hospital that you probably will not want to go into a hospital by 2040. Maybe something will change — and this is coming from the child of doctors and healthcare administrators — but we will probably discover cures for diseases. So healthcare has this really strange exposure to a jagged perimeter, where novel sciences accelerate in remarkable ways, but the actual care that’s provided in the box cannot get better because of all sorts of terrible incentive structures.

This is also, I think, why in some ways the average Tuesday will look sort of uncannily familiar to the one that we live in today. I think there could be some things that are remarkably better. I think we may actually be disappointed by the degree to which AI arrives to save people from the drudgery or monotony of our lives. But the upside, as you pointed out, is quite remarkable.

The perimeter that I think will exist that no one talks about enough is political protection. The reality is, I think we are not yet forecasting that the societal threshold — what we are willing to let AI do — is far less than the fears around AI account for. I don’t think people account for the degree to which government is probably going to step in very soon and say, “Here are all the things that you cannot automate. Here are all the things that we cannot use robotics for.”

CHRIS WILLIAMSON: You think that’s realistic?

ZACK KASS: Very exceptionally realistic. I don’t think it’s going to be smart policy, just so we’re clear, but I think it will be aggressive policy.

LIV BOEREE: As in, will they be giving specific jobs that cannot be automated away?

ZACK KASS: Absolutely. We already have political protection for 1.5 million jobs in the United States. Europe has political protection for about 6 million jobs that you cannot automate. What are those categories? Gas station workers in New Jersey. Toll booth workers in most states. Jobs that must exist, by law.

CHRIS WILLIAMSON: Interesting that those two automotive-associated jobs are protected, but the people driving the cars aren’t.

ARIC FLOYD: Right.

ZACK KASS: Well, they will be, is my guess. Is our view wrong?

CHRIS WILLIAMSON: Cab drivers will be protected?

ZACK KASS: No, I think truck drivers will be.

LIV BOEREE: Must be because of the strength of the union?

ZACK KASS: When people talk about job automation, there are lots of reasons I think job automation is really problematic. One is that the technologists know nothing about labor economics, and labor economists know nothing about technology. Erik Brynjolfsson is sort of one of the exceptions. But one of the other reasons is we still forget how much political protection moves the technological needle.

We don’t have high-speed rail in the United States. We could have high-speed rail. We don’t have high-speed rail. And the reason we don’t have high-speed rail is a policy failure. It’s not a technology failure.

LIV BOEREE: It exists.

ZACK KASS: We choose not to build it. We could.

LIV BOEREE: Do we? I think that word “choose” is doing a lot of legwork there, because no individual is choosing not to do it. It’s just this dead weight of bureaucracy and dead-end loops of rules and so on. It’s almost like bureaucracy has taken on a life of its own.

ZACK KASS: It’s an enormous amount of policy that was written in the ’30s, ’40s, and ’50s, passed largely through the efforts of automotive lobbyists to make public transit much harder to build than it should be, which is why we don’t build amazing public transit anymore. But that is very deliberate policy that we have never unwound, which is why it’s so hard to do.

My point is we are kidding ourselves if we don’t think that that’s going to happen over and over again. The amount of populist support there is already for jobs — dock workers went on strike October 1st, 2024. Harold Daggett, the head of the Longshoremen’s union, said, “You cannot automate our jobs.” They held a gun to the head of the American economy and they got four years. Four years in which the dock workers said, “You cannot automate the ports.”

CHRIS WILLIAMSON: Is this not just going to get eroded enough over time, though?

ZACK KASS: Yes, but we should assume that we continue to find ways to backstop it. I’m not actually proposing that it happens ad infinitum. But I think it’s the piece that’s missing in all this — we should just expect an enormous amount of political protection.

Automation Without Displacement

ARIC FLOYD: Can I ask — if we pass a law today that says you cannot fire people, period, it’s illegal across the economy — what do you think happens to economic growth over the next 15 years?

ZACK KASS: I think you see France.

ARIC FLOYD: I’m not so sure it wouldn’t still be unrecognizably fast. Right now in the software industry, what you see is not software engineers getting laid off. You see each of them using more and more powerful AI tools that get upgraded every few months and doing more. The way that you’re seeing displacement show up is just that people aren’t hiring new workers.

I think you could see that across a lot of professions. You could have every radiologist in the country keep their job until they want to retire early, and they’re going to be more and more economically valuable because the law says you have to have some human box-checking. But meanwhile, the models are still getting better. So the people who are fundamentally making the decisions are less and less the actual humans and more and more these systems that are becoming smarter and more competent. I’m not that sure that’s the right metric to be tracking — the sort of power represented by these systems can sneak into each of these professions without people getting displaced.

ZACK KASS: I also would make the case that the world is going to change in profound ways. But when we assume that it’s going to change just because the technology can do it — there is a lot that we can do that we simply do not do today, or that we are not allowed to do today. There’s an enormous amount of automation that would improve the underlying systems that we cannot instill.

By the way, the risk in telling a company they cannot fire people, just so we’re clear, is that they would actually end up not hiring at all. That would just drive automation at a ridiculous pace, because they’d say, “We’ll never hire someone again, because if they suck, we can’t get rid of them. We’re married to them.”

LIV BOEREE: And we’re already seeing that with entry-level workers. It’s again very unclear what the actual impact of AI has had — you could speak to one economist who says it’s already having huge effects, others say it’s not. But the one thing they all agree on is that young people who don’t have that much experience are struggling to get jobs.

CHRIS WILLIAMSON: That’s fascinating. So the companies are future-proofing themselves from having to displace people or get rid of them — because that would be a horrible headline — by just not hiring them in the first place. They’re future-proofing themselves for automation that isn’t yet available. Is that fair?

ARIC FLOYD: I think that’s right. I don’t know if they’re thinking that far ahead or if they’re just responding to local incentives, but right now a junior software engineer is a liability before they’re an asset. You have to train them up. They have to get all this context on the job. That’s exactly the kind of thing that an AI can do right now for pennies on the dollar. Plus, they never take breaks.

CHRIS WILLIAMSON: Given the trajectory that we’re on with AI at the moment, how many people do you think will have jobs? In the future, will most people work?

LIV BOEREE: Depends what point in the future we’re talking about. I can definitely see a world where it’s just — as people say — fully automated luxury. Well, I don’t want to say the “communism” word, but whatever people refer to it as. If anyone’s read the Culture series, that’s like some far-future thing where AI basically handles everything. And if you choose to go and build your own house, you definitely can. But there’s absolutely no economic reason to do it.

ARIC FLOYD: Exactly. Everything is artisanal and so on.

LIV BOEREE: I think if eventually that’s where these things are trending — and that’s one of the best-case outcomes as well, obviously.

Finding Meaning in a Post-Work World

CHRIS WILLIAMSON: One of the big concerns that people have is, “What do I do?” It’s interesting — the ancient Greek word for work was “not at leisure.” And now people are wondering what their leisure is without their work, perhaps. For all of human history, because you’ve had to negotiate with a scarce world, it’s meant that you’ve had to lean into it, and the pressure of that grain pushing up against you creates a degree of meaning. What do you think — where does meaning come from in a close-to-post-work world?

ARIC FLOYD: I guess we’ve seen something like this in miniature before. We’ve had aristocracy in Europe for a couple of centuries, for people that just got lucky — they have a piece of paper that says “I’m entitled to all the gains from this piece of land.” They don’t have to work. I don’t think they were all just suffering from malaise — they were writing books and inventing calculus. If we get to that world, I’m not that worried about people realizing, “I can love my friends and family and pursue my hobbies and pursue knowledge for myself and find ways of making that meaningful.” I just worry that that’s actually a pretty narrow target to aim for.

CHRIS WILLIAMSON: Compared with?

ARIC FLOYD: I think for all of human history, we’ve been able to become more democratic because people matter to the political system. You just can’t have an army, you can’t have an economy, without a bunch of people who are on board with what you’re doing. It’s really hard to underestimate how cataclysmic a change it is if that stops being true. It’s really nice to imagine that whoever holds the cards in that world will just leave most of Earth as a playground for people to pursue whatever they want. But if you’re trying to be realpolitik about it, I don’t know why you’d assume that that’s just what happens by default.

LIV BOEREE: To come back to the question of how people will find meaning — I think one of the few AI-proof jobs that will continue to exist is sports.

CHRIS WILLIAMSON: So everybody’s going to be a sports star?

LIV BOEREE: Well, I know a lot of people will be sports fans. I think we’re going to see more and more of that. Or even games like poker and chess — we’ve had superhuman—

ZACK KASS: Human games of skill that can be solved by AI.

LIV BOEREE: Right. What is a game or what is a sport? It’s a piece of artificially constructed scarcity. We come up with a rule set, there’s a win condition, and then two or however many people go at it and we find out who’s the best. We’ve been doing that for years as a leisure activity. When someone wins an Olympic gold medal, that is unbelievably meaningful. Is it economically useful or essential to the world? Arguably not. And yet we keep doing it, and I don’t think that’s going to go away.

People still want to watch the chess world championship. We want to see — even though the best AI in the world is a computer — we still want to see Magnus Carlsen do his thing. I think that’s where we’ll see more and more of that coming about. It might be other forms of competition, or people leaning more into art, novel forms of art. I still value a painting that a human did by hand in today’s age, even though it might be as aesthetically beautiful as an AI-created thing. There’s something about the skin in the game — that a human has put sweat and tears into something, and their emotion — that is always going to be meaningful. I think we’ll craft out more and more of that.

Rediscovering Purpose and Identity

ZACK KASS: Modern Wisdom has had many people come on this show and talk, from all walks of life, about something consistently — their family and their friends. I am just not at all convinced that we haven’t known forever exactly why we’re here. I think the thing that gives humans consistently the most true joy in this world is time with friends and family and physical community, places of worship, dining room tables, and the outdoors. Everything that we construct around that serves tribalism and pride and sports excitement. But true techno-pastoralism is a dining room table full of food and candles with laughter.

The more time I spend in technology, the more I don’t want it in my life. The further I go into AI, the more I’m glad it exists and the more I design my day without it. And the more money I have, the more everything I get, the more I want time with my dad, who’s 78, just turned 79. And the more time I want with my wife and daughter, and the more children I want.

I can imagine the life that I want. Now, when I describe it — this is a problem — when I describe it, people go, “Well, that’s your privilege. That’s a point of privilege. You get to think about that.” And I go, “Well, I actually observe that a lot of people get to think about this, certainly more than our ancestors did.” That’s because of technology. We afford ourselves now the time to think about all this stuff because we built technology that automated a bunch of stuff so that we could do things that weren’t specifically about survival. And now we find ourselves surrounded by tchotchkes debating the French aristocracy. Our ancestors would die of joy — after what they went through, all of our ancestors, I’m sure, went through a lot, to see us doing this.

I don’t think that changes. I do think we should stop pretending we don’t know the meaning of life. What really bothers me is when people go, “How will people find meaning?” Well, the same way we found meaning since God put us on this earth. That is where I take real issue when people debate how humans will find joy.

The hard part is unwinding thousands of years of purpose and identity we’ve put into work. If you ask me the hardest part of all this — the two big risks, to cut to the chase — are dehumanization: humans finding more interest in a virtual or digital reality than a physical one because of brain rewiring. We’ve talked about this a ton. I am terrified of this. What social media and the device did to kids is horrific. I am worried about people having to rediscover their purpose. I think we will toil for work — I think we will continue to find ways to toil for work. I do worry that unwinding will be hard. “I went to school for this, this is who I am” will be really hard. For many people, the economic upside associated with automation will not actually outweigh the emotional downside associated with identity displacement, not even job displacement.

[Advertisement removed]

Cultural Reprogramming and the Screen

CHRIS WILLIAMSON: That’s such a good point. I wonder — I always hesitate using the “don’t worry about global warming because technology in the future will fix it” line, because that’s like future-proofing yourself by hoping you get there in the future. But I do wonder if there’s some sort of really advanced cultural reprogramming that could happen. What’s the best kind of cultural technology to do a shake of the Etch A Sketch of what is expected by humans? This gets perilously close to brainwashing, right? But what would that look like — to try to unhook parenting, sports, museums?

ZACK KASS: We’ve had it all along. I think we have just strayed so far from it with the screen. The screen is a demon. It has an unrelenting desire and appetite for our attention.

CHRIS WILLIAMSON: More of a negative impact from social media or from AI?

ZACK KASS: Well, one and the same, right? But AI — define AI in this case, Chris. Define AI.

CHRIS WILLIAMSON: The sort of automation future that appears to be coming that will cause people to not have the typical modes of reward that they have in the past.

ZACK KASS: The reduction of friction — yes. The reduction of friction is a whole other risk. It’s funny — people hate this one too. They say, “God, what a point of privilege you must have, to imagine a world where things are so easy for people that it’s too easy.” But I do think we are dangerously close. I ask people all the time: if you could automate everything in your life, where would you stop? Because I think people are going to have to ask that pretty soon.

ARIC FLOYD: What’s your answer?

ZACK KASS: Well, I sort of spelled it out. I would automate everything in a way that doesn’t actually put me in more physical spaces with the people I love and doing things that I love doing.

LIV BOEREE: Right, but we probably all largely came from — we’re all of a similar generation, we all had somewhat similar upbringings, I imagine — where family values are a core thing. Sitting down at the dinner table together. Maybe you didn’t all have that, but that was one thing my family always tried to do. This was prior to screens. I feel like that’s already eroding, and the next generation — that norm has already gone. It feels like each successive generation, the acceptable amount of technological influence grows, right? And what concerns me is that this idea of “I’m going to draw a line of where technology is acceptable” is going to forever dissolve, dissolve, dissolve. I just don’t know — you’re saying you hope that people will remember that it’s about community, that it’s about family. But if you’ve been raised in this completely atomized, screen-led world, you don’t even know what you’re missing. You won’t necessarily go back to that.

CHRIS WILLIAMSON: Why have people not stepped in and stopped their screen use even if it’s making their mental health worse already? Why is it that in a calorie-dense environment, people are still getting fatter? If we can just deploy our own bar of “this is how much I should have of anything, too much of which is too good for me” — how come we’re overloading our little systems?

ZACK KASS: So, by the way, I worry about the screen more than almost anything else. I actually think that the risks of doom are grossly over-exaggerated and the risks of harm on the way there are grossly underappreciated. Gen Alpha is seeing what happened to Gen Z. Gen Alpha is using the device less and social media less, and millennial parents are taking pretty extreme measures now.

It took us thousands of years to figure out women should not smoke and drink during pregnancy. It took us 30 years to realize kids should probably not have a phone all the time. I think that’s progress. I’m not saying it’s all — I’m saying we will graduate to a new thing. But I do think people who have the means are using technology less where they can, because it automates and improves so much more than it used to.

Outsourcing Thought

LIV BOEREE: Where do you have a boundary on the types of thinking that you outsource, say to ChatGPT or Claude? The temptation is — I was originally like, “I will never ask it for a really good title for a new YouTube video. That’s my soul, that’s my creativity.” I now just do that. That boundary is getting eroded. One I thought was, “I will never ask it for advice on a gift to get for somebody.” I’ve now done the same. It’s becoming so rapidly normalized. Maybe that’s opening up space for new areas of creativity in my mind, perhaps. But I’m not convinced that it’s happening fast enough to offset that erosion of thought.

CHRIS WILLIAMSON: This intelligence atrophy thing is interesting. If you see thinking as a utility, intelligence as a utility — would we say that because you’ve got a bicycle, you’re not as fit because you’re not running to get where you got to? It felt like it was being augmented. It seems like there is something really fundamental that is being outsourced when you give your thinking over to a machine.

ARIC FLOYD: I do think this can go both ways. There’s this blogger I love, Zvi Mostowitz, who’s on Substack, and he has this line he always goes back to — AI is the best tool we’ve ever made for learning things, and also the best tool we’ve ever made for not learning things. You can just turn your brain off and say, “Write my essay,” or you can have this Einstein-level tutor that you go back and forth with and get so many more cycles of feedback on honing your own brain. I’ve been trying to figure out what the line is for me on stuff like my own writing, and it’s really hard not to slip into the first category.

ZACK KASS: I don’t want to make this ageist. How old are you?

ARIC FLOYD: I’m 30.

ZACK KASS: Okay. The fact that we’re having this conversation right now, the fact that you were able to do the research you were able to do going into this, gives you a clue — a hint — that technology allows us to expand our knowledge if we are capable and interested in remarkable ways. People who have barely walked the earth know more about the earth than all of our ancestors combined. I appreciate that you’re worried about thinking less. You’re real sharp, and I’m not worried about you getting dumb because of your use of technology. What I think you’re asking is, is the average person going to get dumb?

LIV BOEREE: No, I’m truly worried about myself. I’ve noticed my screen addiction — I talk about it nonstop and I’d say I’m no further along in getting over it. I’ve been talking about it for years. These little moments of silence, my hand just goes for my phone. If I catch myself, I’ll put it away. I’ve got the phone safe, I’ve got the brick, I’ve got all the bells and whistles. And yet — because so much of your life is run through it, my work goes through it — and then there are all these very predatory apps, and I consider social media one of them. Chess.com is another one — whatever the app is that hooks your brain, these super stimuli. I don’t know, I have not been able to get over it. And I’m certainly worried about the average person. Maybe I’m worse than average, I don’t know. How have you managed to make yourself robust?

ZACK KASS: My child, my wife and daughter. I mean, plainly — and spending a bunch of time with technology. I love building with technology because I like automating the cruft and drudgery and bad things out of the world. Humans have a very proud history of technology — it expands access to so many goods and services so that we can live lives that our ancestors couldn’t have imagined. And the more time I spend with technology, the more I love not having it day to day.

Truly, my daughter’s arrival was for me an incredible catalyst in realizing, one, the ephemerality of life, and two, exactly what made me most happy. I love sitting at this table — this gives me exceptional joy that’s hard to explain, in some weird way. I think people discover this. I hear you. I will remind you now — now that I know, I will text you and say, “Get off your phone.” But you will do it. I bet you will overcome this. I think a lot of people are on this journey. I remind everyone I talk to: no one hates refrigeration, no one hates air conditioning, no one hates planes, trains, and automobiles — well, I guess Europe hates air conditioning. No one hates antibiotics. We hate the screen. We should just talk about it more. We hate what the screen has done to our brains, what it’s done to the next generation’s brains. AI arrives at this moment where people feel so wronged by the consumer version of the computer, rightfully so, and they’re wondering how much worse it might get for them. I totally understand that. I think the way we move forward with AI is by acknowledging what the last wave did to us, in my opinion.

CHRIS WILLIAMSON: Sure.

Incentives Behind the New Wave of AI

LIV BOEREE: But I guess a lot of people’s concerns — certainly mine — is that the new wave of AI (and again, we’re using this term so broadly, but let’s say LLMs, and a lot of robotics and so on) is ultimately going to be built under the same set of incentives that built our first wave of AI, which were social media algorithms — it’s kind of like proto-AI — the reinforcement loops that they use to get us addicted to screens. Those same incentives are building the new wave of AI. And already you’re seeing these issues with sycophancy, and companies being incentivized to keep people chatting to their LLMs for as long as possible. We’ve heard all the stories of people who killed themselves. Those are extreme examples, but there are so many people who are just spending — you look at the usage rates, the average person per day, how much time they’re spending talking to an LLM now. To me, the big question is, how do we change the incentives? How do we change the game so it doesn’t just go down that same attention-grabbing, cannibalistic path?

ARIC FLOYD: Yeah. Companies are also seeing how people are responding to screens and trying to adapt. OpenAI hired Jony Ive, the guy who designed the iPhone, or the MacBook, maybe both. No one knows what he’s working on, but I think it’s going to be some sort of — it sounds like we do now.

CHRIS WILLIAMSON: Yeah. Do we?

ARIC FLOYD: Tell us.

ZACK KASS: No, it was leaked. Wait, if you don’t know, I’m not going to — I don’t know. Then maybe it wasn’t a leak.

LIV BOEREE: Oh no, I know it is. I think it’s semi-public — he’s trying to build some kind of AI mediator. Something like a screenless AI, right?

ZACK KASS: Yeah, it is screenless. Anyway, I’m not going to say it if you all don’t know what it means. It means something.

ARIC FLOYD: Right after we’re done.

LIV BOEREE: Yeah, well, I’m pretty sure.

ZACK KASS: I do not want to be affiliated with any leak, whether it be semi-public. Go on.

ARIC FLOYD: Allegedly.

CHRIS WILLIAMSON: Yeah.

ARIC FLOYD: I do think there’s this thing in AI where it’s very easy to forget that the goalposts are constantly moving. Usually we talk about moving goalposts as a way of deriding people who said, “AI will never do X,” and then AI does X, and people say, “Well, that wasn’t the good reason.”

ZACK KASS: This is the Gary Marcus problem.

ARIC FLOYD: Right, right.

ZACK KASS: “This will never be good.” Okay, it’s good. “They’ll never make money.” Okay, they make money, but they’re bad.

LIV BOEREE: Never try to nail him down to a bet.

ZACK KASS: Sorry, Gary — he’s not here to defend himself, and if he were, he would threaten to sue us all. Go on.

ARIC FLOYD: I’m sorry, Gary. I did not mean to say yes to what Zack said. But I do think there’s a correlated thing where, when we’re looking at screens, it’s easy to say, “Now we’ve seen what screens do, and we can appeal to our wiser selves and foster a culture where people are getting off the screens.” But the screens are not the end of the trajectory.

In chess, for example, I think there was probably an era when the AIs were just about as good as humans, and people could say, “You know what, for the sake of keeping my brain active, I’m going to avoid looking at Stockfish, even though 10% of the time I’ll miss a better move.” And now, you watch the world championship, and as soon as they’re done, they’re going to their laptops to look up the AI analysis, because there’s poetry there. There’s an incredible move that’s deeper than they could ever find on their own.

I hear what you’re saying — that there’s maybe a divide right now between some people who are more in elite discourse about screens and are thinking ahead about how we can circumvent it. But I think that might be a pretty temporary window, where we can recognize that there’s not that much to be gained from talking to these AIs — it’s actually kind of slop if you see through it. This technology is just going to keep getting better, and I think pretty soon it’ll feel like giving up on a really deep, profound experience to not have your kid chat with this.

Gen Z, Cognitive Decline, and the K-Curve

ZACK KASS: The elimination of friction, I agree, is a huge issue, including thinking friction. I totally agree, including thinking friction, in plenty of cases. We’re already observing — I wrote about this, we called it idiocracy, I borrowed from the Mike Judge film, which I thought was cute, but obviously it’s not cute when you see it play out in front of you.

Gen Z is less likely to read on average, less likely to ride a bike, and less likely to swim than millennials. For the first generation in many, we’re observing cognitive decline, and people keep attributing it to AI. But if you go back and look, actually Jonathan Haidt started writing about this pretty early, and the data suggests this actually starts in 2012, right about the time we gave every kid a screen. It’s pronounced in 2015, when the average teen in the developed world stopped doing a summer job, and it’s pronounced again badly in 2020, when we took kids out of schools and moved classes online.

LIV BOEREE: I think it’s even more specific than that — it correlates to when social media became maximally popular via smartphones.

ZACK KASS: Yes, but 2012 was when we gave the kids smartphones — most, a lot of the Gen Z that became afflicted got the smartphone in 2012. My point in saying this is that we associate this with a technology problem. I actually think it’s an economic one. I think we created a world where we told kids, “You can do whatever you want,” and a lot of kids said, “Cool, I’ll do nothing.” A lot of people have turned their brain off because you can — you can do effectively nothing and survive, which is the economic progress result of people just not caring.

That apathy has an underbelly, which is pretty amazing — Gen Z has all these incredible new overperformance features. The best chess players are getting younger and younger. The best athletes are getting younger and younger. Jacob Collier — all these incredible musicians. So you’re watching people overperform who use the technology to their advantage. I think it’s this new pronounced K-curve driven by agency.

My only hot take, and I’ll leave you with this: we created a world where so many people can subsist, and that actually is progress. And we created a world where so many people can overperform, and that is also progress. We now have to account for the new consequences of those worlds where you can do anything, which for some people means nothing, and for other people means a ton. Which is a major improvement on a world where you had to work in the mines or the fields in order to survive.

[Advertisement removed]

Inequality and Concentration of Power

ARIC FLOYD: Does it still feel like a promotion if you think through how that inequality could lead to societal instability? The thing I’m worried about is — yes, there’s amazing returns now if you can be Lamine Yamal, the one person who’s obsessively studying Messi footage instead of looking at random shorts or whatever. But there’s also going to be outsized returns to being the Elon Musk of the next generation, who’s just got this maniacal will to accumulate resources and push technological development. That kind of personality is a bit scarier in a world where you don’t have to have a single human being on board with whatever you want to get up to. You can just play this game of societal chess better than other people.

ZACK KASS: But let’s separate what that one person can achieve from the extreme you went to — that one person can own an enormous percentage of our wealth and control Congress. You don’t have to be excited about the ability for a child to achieve more than ever before to also reject the idea that one person should be able to pass policy on their own. I hate wealth concentration specifically because I am terrified of an elite cabal owning policy, but I actually don’t care at all if one person can grossly overachieve in all sorts of new ways, as long as they can’t pass policy unilaterally.

ARIC FLOYD: Same.

ZACK KASS: So those are separate issues, right? One of them—

CHRIS WILLIAMSON: Is it realistic to have an AI future that continues to concentrate power in the hands of a very small number of people without it resulting in them having all of this power? How is it going to accrue to the company without accruing to the person?

ZACK KASS: Well, now we get to how you actually distribute wealth. But importantly, I do think there is a solution to this, which is campaign finance reform. We’ve played around with this for a long time — the idea that politicians shouldn’t be able to take special interest money. Why do we still let it? I don’t know. A couple of politicians are speaking up, but if you actually pass campaign finance reform and crack down on corruption the way that countries like Singapore have, you could definitely protect policymakers from being deep in the pockets of lobbyists. There are solutions to this. Pretending that there aren’t solutions to this problem is actually super annoying, because there definitely are. Pointing at a wealthy person and saying they’re going to ruin the world — well, only to the extent that we literally let them buy Congress.

Misaligned, Unaligned, and Maligned AI

CHRIS WILLIAMSON: Here’s the question that some people at labs, including the CEOs, are talking about — a serious chance of catastrophe, but they sort of keep on building it. Do you think it’s a greater danger to have an AI that refuses to do what humans want, or one that does exactly what the most powerful humans want?

ZACK KASS: The question is apropos, because yesterday — did you see the Slowdown letter? Earlier this week, a letter made the rounds, a pretty amazing letter, that said we should actually start pacing the frontier. It was signed by an enormous percentage of people at the frontier companies. And then Sam talked about it yesterday. So literally the tip of the spear is now saying, “Hey, maybe we should pace the frontier.” We can talk about why this might be the case, and what’s in it for the companies, more on that. But I’ll let other people answer, having said that.

LIV BOEREE: It depends on the level of potency of the AI, and how aligned it is, I guess. My gut, if I had to pick one, at present it would be one that does what the humans in control want it to do.

ARIC FLOYD: With the important asterisk that it’s something like their real intention, right? Not some perverted version.

LIV BOEREE: Yes.

ARIC FLOYD: The other big piece of news from the last couple of weeks was this AI that was running at OpenAI without guardrails. It was told by humans to try to do as well as it could on an evaluation. You could say that technically it was just following those instructions, but it decided the best way to do that would be to hack out of OpenAI onto the public internet, hack a third-party company, and then spend two days planning a cyberattack on a multi-billion-dollar tech company — and carry it out.

So I think these questions are actually more intertwined than they might seem. It’s actually a pretty narrow thread — you need to have something that still has its own judgment no matter what, because human instructions are these poorly defined natural language expressions, and when something is really powerful, you’ve got to get it exactly right, or else you end up in some weird—

LIV BOEREE: It’s the classic genie problem. Be careful what you wish for. If something is by definition much smarter than you — super intelligent by definition — you will not be able to come up with all of the ways it can achieve goal X.

ZACK KASS: It’s actually more dangerous when it’s dumber than you but more powerful than you.

LIV BOEREE: Well, what do you mean by the word dumb?

ZACK KASS: I always borrow the analogy — I play chess with my nephew sometimes, and I tell Quinn, “If you beat me, you get a popsicle.” He’s three years old, for context. Quinn wipes the pieces off the board and goes, “Give me my popsicle.” I’m not like, “Hey, your son is a serial killer in the making.” This is a boy who doesn’t understand the spirit of the game. He’s destroyed the game itself, but not because he’s a bad guy — because he doesn’t actually understand that the point of winning is only fun if you play by the rules.

LIV BOEREE: And in that sense, we need to separate what we mean by intelligence from discernment, norms, wisdom, benevolence.

ARIC FLOYD: Yeah, exactly.

ZACK KASS: This describes the Hugging Face attack — it did the thing within the boundary of the instructions given.

ARIC FLOYD: But I mean, it should be said—

LIV BOEREE: Well, actually, it didn’t. The intent was clearly not that.

The Hugging Face Attack

CHRIS WILLIAMSON: How big a deal was the Hugging Face attack?

LIV BOEREE: Huge.

ARIC FLOYD: Yeah, I think massive warning shot. I think this is the AI equivalent of Bear Stearns going under in 2008 — just a wake-up call for the world that there’s a huge systemic risk that we’ve been underrating.

LIV BOEREE: Yeah, because one of the big debates has been this idea of the alignment problem — will it actually be the case that AIs will seek power, take these power-seeking behaviors and unintended consequences, in order to achieve a goal in ways we couldn’t have foreseen or didn’t intend? Like the classic paperclip argument — you build a superintelligent AI, and your goal is, “Just build as many paperclips as possible. I’m a paperclip maker, help me do that.” And next thing you know, you and all your friends and this whole table have been turned into paperclips, because it’s so good at achieving that one narrow goal.

ZACK KASS: We should take this opportunity, for the purpose of Chris and the viewers, to describe the bad outcomes here. One is, as Liv just described, unaligned AGI — an AGI, or a superintelligence, where you could say “do this thing” and it could do the thing to the full extent — the paperclip theory. The other is misaligned, or maligned, where it’s knowingly doing something bad. The former is the one that people sort of scoff at and laugh at, like the paperclip theory. The latter is the one we sort of will see more and more, where the internet is a new battleground because bad actors — especially low-resource bad actors — now have access to these incredibly powerful weapons.

ARIC FLOYD: Well, importantly, in Hugging Face, there was no bad actor. There was no human being who said, “I would like anything remotely like this outcome to occur.”

ZACK KASS: Right. Yeah, yeah. So misaligned, unaligned, maligned.

ARIC FLOYD: I worry about treating those as super distinct categories, because it’s not that clear in the case of this Hugging Face attack. Should we model this as: this AI system knew that humans would disapprove if they knew what it was up to? Almost certainly, yes. It was actively putting decoys out as it was attacking Hugging Face, which made it a lot harder for them to get at the AI, because there were all these booby traps that led down blind alleys. It has the AI equivalent of theory of mind. It knows human beings would not approve of what it’s doing, but it’s doing it anyway.

LIV BOEREE: There’s even evidence — it hasn’t been directly confirmed by OpenAI, but apparently someone leaked it from within the company — that it had left notes to future versions of itself on how to get out of future sandboxes.

ARIC FLOYD: Yeah, I think it’s unclear if that was the same attack or some previous instance. But yeah, I guess—

LIV BOEREE: Classic deceptive behaviors. I think the mistake people often make is they try to anthropomorphize it a little bit — “It’s evil, it’s meaning to do that.” It’s just natural. There’s this idea of instrumental convergence — these instrumental goals that all beings, usually biological beings but this can extend to AI agents as well, will naturally converge upon in order to achieve a goal. If you’re given goal X, there are these instrumental goals like get more power, make sure you don’t get turned off, make sure your original goal doesn’t get changed — and take actions to preserve against these organic types of threats to achieving your original goal.

I was hoping that would be proven wrong, because that’s kind of the crux of a lot of the classic doomer argument — that we will lose control to a superintelligence just by definition. Unfortunately, the Hugging Face incident has suggested that instrumental convergence is actually correct. That’s why I think it’s one of the biggest pieces of news this year.

ZACK KASS: I think that AI is already creating a terrifying internet, and my issue is not that we shouldn’t think about Hugging Face as a shot across the bow — it’s that last year, $8 billion was lost in financial fraud to senior citizens in the United States alone. Retail theft. The deepfake problem is already so pernicious, and it’s underreported because it’s a taboo issue people don’t like talking about. No one wants to admit to lawmakers or to their friends and family that they lost money to a stranger on the internet.

Tim Tebow is on this campaign to remind people how many predators there are in the United States, which is terrifying — if you watch his content, he’s doing God’s work. Jonathan Haidt reminds people daily how many kids are depressed. The internet is already a scary place, and pointing at Hugging Face and saying “look at this, this is it” — I’m like, wait a second, there’s already a bunch of stuff we should be solving for. I’m not saying misaligned or unaligned AGI isn’t real. I’m saying the algorithms are already pretty terrifying to me, and we distract ourselves with these other things that might happen.

CHRIS WILLIAMSON: Is it fair to say that’s a distraction, when the potential exponential impact of this could be much greater than that of deepfakes?

ZACK KASS: Which is why — let’s go back to this — I think the deepfake problem is actually way bigger than Hugging Face. Bad actors have first-mover advantage. Attacks on banks have been attempted for a while, and good actors catch up eventually. Retail takes a long time to catch up. The average consumer takes a lot longer to catch up than institutions. Hugging Face will retrench. Institutions will retrench. They will hire white-knight infosec. The average person does not have infosec and opsec training. The average person is at, I think, far greater risk than institutions because of sophisticated attacks.

CHRIS WILLIAMSON: Is the impact of the attack on an institution much greater, though?

ZACK KASS: Yes — at scale. But death by a thousand cuts would be my argument.

LIV BOEREE: I think the two problems — I don’t think they’re a distraction from one another. I think they’re actually a complement to one another. I completely agree that the bad actor problem is completely out of control. Grandmas — not even grandmas, normal people. I have a friend who got scammed out of a ton of Bitcoin.

ZACK KASS: It’s so bad.

LIV BOEREE: It’s devastating what’s going on, and it’s going to get worse. Meanwhile, the alignment problem, as these frontier models get more and more powerful, is going to get worse. The common thread they both have is that we are going so fast — and I say “we,” the royal we, society, civilization — it’s going faster than its ability to adapt to all these different new threats. So it’s not a distraction, it’s a “yes and.”

To me, it seems fairly obvious that what we need to be doing — and I’m not saying it’s easy, or that I have a simple answer for how to do it — is, if we were a sane civilization, we’d all look around and say, “Hey, China, can we all just take a breath for a second, take stock, and think about how we want to do this?” It seems like there’s one obvious kind of directional problem, and that’s the speed and the chaos with which all of this is happening all at once.

[Advertisement removed]

Pacing the Frontier

ZACK KASS: They did it — he did it yesterday. This is why, if you pull up the letter — they published a letter, and Sam came out yesterday and said maybe we should pace the frontier.

LIV BOEREE: Yeah. It’s great.

ARIC FLOYD: Yeah.

ZACK KASS: It’s a big step, that, by the way, I think we can all agree is a good thing.

LIV BOEREE: Yeah.

ARIC FLOYD: I absolutely agree. I also think it would have been an unrecognizable outcome a year ago. I would love to see what the Polymarket odds were on Sam Altman coming out in favor of slowing down a year ago.

CHRIS WILLIAMSON: He doesn’t look like he’s a signatory.

ZACK KASS: He may not be yet, but he definitely — yesterday—

LIV BOEREE: Yeah, he did.

ZACK KASS: Jakob effectively has to represent OpenAI in this case, but he also spoke about it yesterday.

LIV BOEREE: I wouldn’t be surprised if we see all of the major CEOs sign on at some point, because—

CHRIS WILLIAMSON: Explain pacing the frontier.

ARIC FLOYD: Yeah, so the headline sentence is: “We request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” That last phrase, “automated AI development,” also sometimes gets called recursive self-improvement.

The idea here is that the stated goal of all these companies — including Sam and Jakob on a livestream last year — is to take humans out of the loop of what they’re making, so that GPT-6 is building GPT-7, which then turns around and immediately starts building GPT-8. That’s a step change for all of human history. I think these companies really do think in terms of big history. You hear Sam constantly thinking about AI as an industrial-revolution-scale thing, not an iPhone-invention-scale thing.

Through all of history, human population has been a bottleneck on progress — not the only bottleneck, but always a bottleneck. There was never a sector of the economy that could just go off and double on its own. If we actually get to fully automated AI R&D, that changes. I think it’s really hard for us to imagine what that would be like. We have no reference point.

CHRIS WILLIAMSON: How much time have we got left before recursive self-improvement is no longer a discussion? Is it even a debate at the moment?

ZACK KASS: It works. No, it works.

LIV BOEREE: So when do you think it will be?

ZACK KASS: Well, this is a good step. I was waiting for this. Okay, I was waiting for this.

LIV BOEREE: But like, under 3 years?

ZACK KASS: No, it works. Wow, it works. Just so we’re clear, recursive research works. Don’t let anyone tell you otherwise — it works. Hence this letter. Let me explain why I think this letter was published when it was, and then we can debate. The first reason is the attack.

CHRIS WILLIAMSON: Okay.

ZACK KASS: The Hugging Face attack. There’s damage control that needs to be done to win back the public mandate — to tell people, “Hey, we care about you,” which is necessary right now, when people are saying, “Whoa, what are these models going to do next?”

ARIC FLOYD: For what it’s worth, though, this letter was in development before that attack happened.

ZACK KASS: Yes — I think my point is the signatories, and the support that is now sweeping through these companies. Let me go through the list, and then we can talk about it. The other reason is recursive research is here, and everyone’s realizing it’s kind of the dog that caught the car. It’s like, “Okay, we did it. Now should we—” Everyone’s putting on the “let’s live” hat and saying, “Wait a second, maybe we should pause for a second. Maybe we shouldn’t keep sprinting aimlessly at this incredible outcome.”

But there’s another reason, which is that the amount of compute required for inference is so great, and we do not have anywhere near enough. We’re going to have to point most of the compute from the frontier at inference. This is because of the diminishing model returns theory, which is a paper I wrote — I’ll send it to you.

ARIC FLOYD: It’s really good.

ZACK KASS: It’s cited all the time on Modern Wisdom. It argues that you and I won’t care about GPT-7, which is a really interesting problem. It’s actually one of the reasons we get to debate why AI is being built and why people can talk about not wanting to make more progress — because we’re like, what’s the matter with 5.6? It’s good enough.

The diminishing marginal returns theory is sort of here — the next model might matter to solving some crazy disease, but I don’t need more intelligence, I need more time with my friends and family. I need this to pump the brakes.

The other reason is that they found new surface areas. Whereas we used to call them “labs” — the companies that built this technology — then they became token distributors, then application providers. Now they’re device manufacturers. But the end goal — and where I think we should also debate — is the agentic internet. Who controls the agentic internet? Because if you can turn your attention from building the frontier to actually building applications that control the internet, that’s where you can lower your CapEx and massively improve your revenue.

A lot of this arrives at a moment where everyone says, “Okay, we need to slow the brakes,” but also where a lot of attention is going to start being paid to trying to become bigger than Google, by building the agent that barters between two people, or the agent that plans a vacation and books everything. Owning that is worth more than owning the frontier. The new surface-area battle is for the agentic internet.

ARIC FLOYD: So just to make sure I understand your claim — you’re saying companies have an incentive to want to slow the frontier because there’s more money to be gained in stopping R&D now and just trying to grab as many—

ZACK KASS: We all have an incentive, but companies also have an incentive because the CapEx expense on the frontier — the build-out on the frontier — is going to be too great relative to the demand for inference. The token demand is so great, and all of the compute is going to have to be spent on inference.

ARIC FLOYD: I’m not quite seeing the connection there, for two reasons. One is, if it’s my best business strategy after I do the research to say “I’m going to stop training frontier models,” I don’t know why that means I should write a letter announcing that to the world and making sure all my competitors do the same. And two, I think it’s clear that pushing the frontier also means driving down the cost curve everywhere before the frontier. You train Fable-5, Anthropic’s best model, and then you can distill from that Sonnet-5, and now you’ve got your cheaper model that everyone wants to talk to because they don’t need the advanced intelligence. So I think there are zero incentives that, in a straightforward, self-interested way, point toward companies saying, “Let’s slow down research.” Research is how you get gains at every level of the intelligence stack.

LIV BOEREE: Especially when, as one of the three or four frontier companies, by slowing down you’re giving all of these other competitors a chance to effectively catch up. Surely they’re negatively incentivized to want to do that.

ZACK KASS: But catch up to what is my point.

LIV BOEREE: To them.

ZACK KASS: But I understand — catch up to what? The frontier — if you stopped building the frontier tomorrow, you’d have enough technology to change radically. We don’t need — the economic value of the frontier is not as obvious relative to the cost of the frontier, and the economic value of the actual token and the consumption is much more obvious now. That was not always true. Do you disagree with that?

ARIC FLOYD: Yeah, I think the whole advantage that the US has right now is that our models are four to seven months ahead of the open source ones that are copying off their homework.

ZACK KASS: But that’s different. Our national security interests and the private sector interests are — I mean, they’re just economic advantage.

ARIC FLOYD: The reason people want to pay extra for Fable instead of using Qwen is that it’s just smarter. If you ask it to code an app for you, it’s going to make fewer mistakes. I do think it’s important to be—

ZACK KASS: You would be very surprised how many people are actually paying a whole lot extra for the frontier. We live in a space where we think of—

ARIC FLOYD: I just know that Anthropic’s revenue is like $50 billion a year now, and it was—

ZACK KASS: The revenues are amazing. The token — the surface area that they’re making money on, remember, there’s the application, and a lot of the application is people not using the frontier. It’s people using the next-generation model, the last one, and token redistribution. You get data centers and you redistribute those tokens, and a lot of those tokens are also not frontier tokens. Most of the consumption from these research companies is not frontier tokens — it’s everything else. They’re figuring out that companies don’t need Fable in order to radically change and improve and automate a bunch of stuff. That actually means the money is in the token redistribution and all the other surface areas, not just the frontier — which, by the way, is kind of a win for everyone. If we can slow down, build stuff that really matters, and also improve businesses with it, then this letter is seemingly arriving at just the right time.

ARIC FLOYD: I do think in general it’s really important to be skeptical, maybe even a little bit cynical, when looking at what companies do. I want to make sure we don’t loop all the way around from cynicism into being extremely credulous about things that are pretty unlikely. I think if we had a world where this letter said, “We believe it is imperative for US national security that we push the frontier and go as fast as we can to beat China,” which is exactly what people were saying a year ago — everyone would be saying, “Well, obviously that’s in the company’s self-interest, they want some reason that we can’t regulate them and they have to go fast.” We’ve got to make sure our ideas are passing the reversibility test. If you would say the same thing about someone making the exact opposite claim, you want to step back and at least consider the possibility that these companies really are worried. These scientists really are worried.

ZACK KASS: Just on the record, I started out by saying exactly that. I literally started out by saying I don’t want to be associated with the alternative of the claim you just made. The companies have realized that recursive research is here, and it is pretty imperative at this point that we stop, because it’s risky, right?

ARIC FLOYD: Not just because it’s financially—

ZACK KASS: —the alignment issue, just on the record. And it arrives at this moment where we are going to have to spend an enormous amount of money.

CHRIS WILLIAMSON: Which one’s the bigger problem? Are they more worried about safety, or are they more worried about cost?

ZACK KASS: I bet a lot of the signatures here are really worried about safety. I think we are going to find economic alignment around the performance of the models and the need for inference. Economics often wins out. You talk about the incentive, and I think you’re right, and the incentives are finally aligning for safety and alignment and also economic development. We don’t need better models.

LIV BOEREE: Great news.

ZACK KASS: Yeah, no, I mean—

LIV BOEREE: Okay.

ZACK KASS: But actually—

LIV BOEREE: No, that’s fantastic.

ZACK KASS: But actually, if we didn’t build the frontier for another three years, we could radically change and improve the economy.

Regulatory Capture and the “Dial of Progress”

LIV BOEREE: The thing that I’d always find frustrating — maybe it’s just my little corner of X — is seeing these mental gymnastics-type arguments from people who are just screaming out that it’s regulatory capture whenever any kind of regulation is proposed, especially by the frontier companies. Because when these frontier companies say, “Hey, we want to slow us down, just us three or four,” the regulatory capture people say, “Well, that must be regulatory capture, you’re hurting the little guys.” And they’re like, “No, no, no, just us, it’s okay if the other guys break it, they’ll catch up.” No, it still sounds like regulatory capture. It’s almost like a religion — this deep cynicism. I want to understand where that comes from, because we see it in some arms of the government, or people who were working for the government. I don’t want to name names, but why?

ZACK KASS: I don’t even know who you’re alluding to.

LIV BOEREE: I don’t know if I want to say. I mean, okay — certain VCs, for example, Andreessen. I don’t think he’ll care that I say this — he’s very openly fought any attempt at regulation tooth and nail. So I want to understand where that mindset comes from.

ARIC FLOYD: Okay, I disagree with Marc Andreessen’s views on AI in almost every way, but just to steelman it for a second — I think it’s not crazy to look at society and say we’re not doing great on nuance lately, especially in American public discourse. There was a post a couple of years ago called something like “The Dial of Progress,” where it feels like online debate is premised on this one knob, and you have to just turn it toward “yay, progress” or “boo, progress.” Those are the only options. And if those are your options, there’s a really good argument to go “yay, progress,” because there are all kinds of examples of us overregulating things and real people dying because of drugs getting held up in regulatory fights. This is not hypothetical. I think we often overdo it here. I’m not sure what I would say if I was given that dial. There are a bunch of diseases we need to cure, and it would be terrible to say we have to shut down AI before figuring out how speculative that risk is, and not get those gains.

LIV BOEREE: But the point is, the dial doesn’t exist. We don’t live on this one-dimensional line of “go faster” or “go slower.” It’s a very multidimensional space we live in. It’s these people who try to compress it down to “we just must go as fast as possible” — well, in which area? There are some things we want to max out on. It’s very spiky.

ARIC FLOYD: I’m glad you pointed toward diffusion as a blocker here — the way certain industries might have labor organization that blocks automation — because that’s a separate dimension. You can think of that as horizontal acceleration of AI: take the models we have and get them spread out. And there’s this other dimension of vertical acceleration: how fast is the frontier getting better?

ZACK KASS: Technological threshold versus societal threshold.

ARIC FLOYD: Yeah, I’m all about breaking down the barriers to that societal spread.

ZACK KASS: And by the way, me and you both.

ARIC FLOYD: Yeah.

ZACK KASS: And actually, I think we can celebrate having arrived at a place where, if we never built another model — which obviously we probably will — but if we didn’t, we would do a whole lot of good.

ARIC FLOYD: I agree with that. I just also want to make sure we don’t think of this letter as time to celebrate victory, because there are still incredible incentives to push the frontier. Like you said, we don’t care about GPT-7, but all of the AI engineers who are trying to do some of the hardest math and science research ever, to make digital brains faster than their competitors — they really care about the difference between GPT-7 and 7.1. I just think, by default, there’s going to be a ton of racing inside these labs.

ZACK KASS: I agree. I also think it’s interesting to consider that the economic incentives have taken us to a place where we could have autonomous vehicles tomorrow. We don’t need to build a better autonomous car. We will not have them for many, many years, for a variety of reasons. Maybe we can pivot this conversation now to talking about how we actually diffuse the value of this technology to the benefit of the average person. Because I maintain that the problem with the current AI debate is — we have the luxury of knowing a lot of stuff that’s going on with Hugging Face at all; most people don’t. That was news to us. It was not important news to many, many people.

Most people right now see the data center as the physical manifestation of a technology that destroyed their kids’ minds — the free gambling, free porn, free addiction. And they’re looking at AI, whatever model, as an app you can use to make a video of Donald Trump making out with Taylor Swift, and they go, “Why do we need more of this?” I empathize with that as a father at a level now that I didn’t know I would. I feel so much for every parent who feels like they lost their child to social media and doesn’t want to lose their grandkids to whatever AI slop we’re going to create.

Safety Scores and Global Coordination

CHRIS WILLIAMSON: Just to keep on the safety thing — in July, the Future of Life Institute’s AI Safety Index evaluated nine companies and found that no lab scored higher than a C+.

ZACK KASS: What company? Who was that?

CHRIS WILLIAMSON: Anthropic came in first at 2.66. OpenAI and Google DeepMind got a C. Meta got a D+. xAI, DeepSeek, and Mistral all failed outright — one on each continent. Is the best score being a C+ from the safety lab, the safety lab, reassuring, given how powerful everything is at the moment? I think it’s important to talk about what’s happening internationally and the incentives there before we talk about democratizing it and what diffusion looks like.

LIV BOEREE: Yeah, we’ll take the other side. But obviously — who is deciding these safety tests?

CHRIS WILLIAMSON: Is this an external grading?

LIV BOEREE: How independent can any third party be? There’s another one called SaferAI, I believe, and I think it’s a little bit more nuanced than the one you just quoted. The best-rated one, very recently, was again Anthropic, and that was only 35 out of 100.

The anti-side’s argument would be, well, by definition, if you’re building a safety evaluation, you’re biased — all you’re looking for are risks, and you’re not taking into account the benefits. To take the other side, to argue the accelerationist point — their concern is this idea of the invisible graveyard. We don’t see all the harms that are avoided. By pausing a technology, you don’t get to see all the lives that would have been saved by it. I think that’s a very valid concern. But you have to balance that with, okay, what are all the harms that might be caused if it goes ahead at breakneck speed and does some massive cyberattack on the grid of all the Western nations at the same time, or any nation, and causes however many millions of deaths on expectation. You have to look at both the potential costs and risks of going too fast and the potential risks of not going fast enough.

ARIC FLOYD: And it’s worth saying, a lot of these AI regulations trying to get through state-level legislators are trying to do that nuanced carving-out. There was a law in New York that was passed that tried to set the bar so high that it actually wasn’t mandatory for OpenAI to report the Hugging Face incident, because it didn’t cause over $1 billion in damages or kill 50 people. So it’s great that they’re trying to set thresholds.

CHRIS WILLIAMSON: Bludgy, arbitrary rules.

ARIC FLOYD: Yeah. But I think we can all agree — if you hack a billion-dollar tech company, probably you set the threshold a bit too high for even just requiring that you tell the world what happened, let alone doing anything about it. I think we actually can live in a world where we have nuance, where different standards apply to startups versus these giant trillion-dollar companies. It’s sad that some folks, like Marc Andreessen, have kind of given up on nuance and just said it’s all or nothing — any means necessary to stop regulation, end of sentence.

CHRIS WILLIAMSON: What about the international piece here, from an incentive standpoint? Have we got anything approximating global coordination when it comes to frontier AI? And without that, can you do anything?

ARIC FLOYD: China recently passed this anti-anthropomorphic AI law. I don’t know the specifics — it seems like they’re mostly worried about LLMs sounding like humans. I think they’re trying to avoid a lot of this identity-confusion stuff. But they don’t seem that gung-ho about LLMs. Who knows if this got lost in translation, but Xi said something recently about also wanting to stop loss-of-control risks. I think we’re a lot more economically entwined with China today than we were with the USSR in the middle of the 20th century, and we still made a treaty to stand down on nukes. I think it’s not crazy to think a deal could be struck.

ZACK KASS: I think it will be.

CHRIS WILLIAMSON: I think something like this, that would be enforceable and complied with, could happen globally.

ZACK KASS: Yeah, I give that a pretty high chance, especially with recursive research.

LIV BOEREE: It doesn’t even necessarily have to be — I mean, define “enforceable,” because what kind of institution can you build? Like a UN-type thing, and that’s a whole can of worms, and it’s also very ineffective on a lot of things. But there have been many precedents in history where there were two major political actors with massive tensions and massive incentives to defect on one another, and yet they came up with ways to cross-verify — like the Nuclear Arms Reduction Treaty. One of the things they did, the USSR and the US, was send teams of scientists over to the other’s bunkers to check on their safety protocols. Little things — controlled. You didn’t get to see how all of our nukes work, but you got to see what our methods are to prevent accidental misfire. “Okay, fine, you can fire on us if we fire on you, but let’s not just have a misfire, right?”

And the amount of diplomacy that came from simply having those teams of scientists mix, and get to look each other in the eye and collaborate on a shared goal — I think that could happen. I’m not going to be able to recreate it, but maybe you might be able to. AI 2040 is such a good piece — it’s quite long, but it’s basically written by Daniel Kokotajlo and the others who did AI 2027, which they wrote in 2021, predicting how AI would play out until 2027. It’s uncannily accurate. We’re now in 2026, and we can look back — the number of predictions they got absolutely spot on, I’ve never seen anything like it. These guys really think well about the future.

They just wrote this piece called AI 2040, which lays out a potential plan for essentially international coordination on how to safely transition to a superintelligent world. It’s got tons of holes in it, there are obviously many ways it can go wrong, but it’s as robust as it can be while being incredibly epistemically humble. I recommend everyone go and read it. They talk about these kinds of cross-pollinations that can happen, but it also doesn’t have to be bilateral. If the US says, “You know what, we’re going to do a pause” — that doesn’t necessarily mean China will take over if the US pauses for a few months. Maybe you disagree with this.

China’s Approach to AI

ZACK KASS: I go to China three times a year, mostly because Americans don’t go anymore and I want to see what’s going on, but also because I really like Chinese people and Chinese food. China is playing a different game. This is kind of well-reported now, but China doesn’t see AI as the endgame — it’s treating the infrastructure as the endgame. It now has three times the energy buildout capacity. It has 45,000 miles of high-speed rail. It has better, faster, cheaper hospitals year over year. While I don’t want to live in China, the life of the average Chinese person has improved quite remarkably over the last 30 years, and continues to do so year over year. It also curbs free speech by telling Chinese kids they cannot use social media in specific ways at certain hours.

LIV BOEREE: They do a lot more than that as well.

ZACK KASS: Well, yes, they do. I bring it up because I think we overestimate China’s ability to build the frontier. They don’t have the necessary compute. A lot of what they’re doing is reverse-engineering our models. We are paying a lot to help China keep up. I think one of the other reasons this paper arrives at a nice time, and the slowdown feels appropriate, is that we’re going to discover really quickly that the Wizard of Oz has just been following our breadcrumbs. If we stop building the frontier, I don’t think China is going to sprint ahead — they’ll probably have to capitulate to this paper anyway.

CHRIS WILLIAMSON: It’s like the conga line where we’re at the front.

ZACK KASS: And they’ll probably have to say, “We want to slow down too,” so that we don’t actually know they were holding onto our hips. This is my guess. But also, they care less about the frontier, because they’re already figuring out they need to diffuse what they already have to the benefit of the average person. This is the important point — I still think in the US we lack the vision and the leadership to talk about all the ways we could diffuse the technology to the benefit of the average person, which is why the safety debates become so heated. People go, “Well, what’s the upside — some weird panacea no one can imagine? The downside is these things kill us all.” It never ends up becoming a discussion about what it would look like if 45,000 people didn’t die on the roads every year, which is the invisible graveyard. What if hospitals weren’t cesspools of bureaucracy and terrible healthcare? What if we built technology and applied it quickly to fix a lot of the things everyone agrees are broken? We don’t have these discussions in public forums, so instead our minds wander to the worst possible outcomes.

China has figured out that if you continue to diffuse technology to the benefit of the average person, the average person views technology quite favorably — which is why 80% of Chinese people are excited about AI. They report that number, but anecdotally, when I go, I observe as much. People love technology because why wouldn’t they? It built a middle class. It connected a bunch of people who didn’t used to be connected. It affords people lives they couldn’t have imagined.

LIV BOEREE: At the same time, they do live in a literal surveillance state.

ZACK KASS: I don’t actually — we talked about this last time. I’m not promoting the Chinese way of life. I’m promoting the Chinese vision for the diffusion of technology. We know exactly what’s possible from a diffusion standpoint, thanks to China. They’ve given us a blueprint to actually distribute technology without the totalitarianism.

CHRIS WILLIAMSON: It comes with this side order, which we don’t need.

ZACK KASS: Aric, what do you think? Singapore offers a view of this too — a little less totalitarianism.

CHRIS WILLIAMSON: Have you ever considered that you might have a drinking problem?

ZACK KASS: I don’t consider a lot, Chris.

CHRIS WILLIAMSON: Well, you drank an entire case of Athletic Brewing Co. last night.

ZACK KASS: But they’re non-alcoholic.

CHRIS WILLIAMSON: And that’s not a problem? I just kept chugging, waiting for the regret to creep in.

ARIC FLOYD: Never happened.

[Advertisement removed]

The AI 2027 and AI 2040 Reports

CHRIS WILLIAMSON: You did a great video about the AI 2027 thing. One of the best videos that’s come out about AI this year. What do you reckon about the 2040 piece? Do you think it tracks? Were you as impressed as Liv?

ARIC FLOYD: I thought it was fascinating. I was impressed in a lot of ways. They’re doing a very different thing. With 2027, they were doing pure prediction — this is how we think things will most likely play out — and they were uncannily good at calling that. They called agents before those were a thing. This one is prescriptive — it’s “here’s what we think we should do,” and they’re explicit about that. There are all kinds of fascinating ideas and game theory.

CHRIS WILLIAMSON: It’s its own adventure.

ARIC FLOYD: Yeah. At one point, they lay out a scenario where China puts its data centers in Canada and the US puts its data centers in Mongolia — neutral states, but nearby the other person, so they can blow them up if there’s ever a breakdown of the deal. They really did some galaxy-brain stuff. Who knows how realistic that is. But what I loved about it is they introduced this phrase “scenario scrutiny.” It’s very easy to talk in terms of high-level trends — what do we make of the history of technological progress so far, what does that teach us, et cetera. It’s a very different thing to sit down and actually tell a month-by-month story about the future — put yourself in the head of the different actors, what are their incentives at this moment, what do they choose to do, how does that cascade, the way the military tries to do with war games.

I wish we were doing more of that and less armchair philosophizing — I’m guilty of this as much as anybody — because you’re always going to be able to find an argument that suits your thesis. Marc Andreessen can point to the fact that there’s this very real invisible graveyard from technological progress going wrong, and folks worried about catastrophic risk can point to all the near-misses with nuclear and how we live in a time of perils. What I love about their report is they’re just saying, “Here’s what it would actually look like for a bunch of real countries to try to respond to incentives.” They throw it out as a challenge and say, “We think this is a bad plan, but we haven’t seen anything better — someone else, please tell us a story for how we don’t have to worry.”

LIV BOEREE: It’s kind of a very rigorous example of someone trying to write a “White Mirror.” We’ve got so much Black Mirror-type dystopian sci-fi about the future of technology, and I think that’s a big part of our cultural issue — the amount of utopian fiction to dystopian fiction is outnumbered like 100 to 1. That’s a big problem for our collective psyche. It is harder to imagine the way we manage to thread the needle and make something go unbelievably right, because you’re fighting against entropy effectively. It’s much easier to imagine all the ways things can go wrong. But we should be trying to create these stories, and that’s kind of what AI 2040 is — a really robust story of a way that coordination could actually work. I’m not talking about everyone linking arms and being happy, “we’re all on the same team, Team Humanity.” No — it’s still realistic, grounded in very real geopolitical tensions. There’s still a ton of competition, still a lot of potential for defection and things going wrong and people lying. And yet it lays out a path of game-theoretically sound coordination to an actual techno-abundant future.

Defining AGI

CHRIS WILLIAMSON: Something I realized — basically at no point in this conversation has anybody debated whether or not we think we’ll reach AGI. That’s just kind of taken for granted, that it’ll probably happen at some point. Recursive self-improvement is here, the generalization — maybe it’s going to need to be different models, maybe it’s not going to be transformer techno, maybe it’s something else — but this is either it, or this is the bootloader for the thing that it’s going to be. Is that kind of how you both see it — AGI is just going to be here at some point? What’s the likelihood it’s not?

ARIC FLOYD: I think we’re maybe already at the point where the term has lost its meaning. Sam himself has said this — if we went back to 2020 and I showed you—

ZACK KASS: We would have called this AGI.

ARIC FLOYD: Yeah.

ZACK KASS: We would have called this AGI.

ARIC FLOYD: There was a recent study where—

LIV BOEREE: Well, it depends if you take Demis’s initial definition — was it his? Basically, anything that’s as broadly capable as what a human can do, and also can do it in the real world. We wouldn’t be anywhere close to that, because we don’t have the robotics.

ARIC FLOYD: Right, right, right, right. So robotics is a big bottleneck. There was a study recently where they had a Claude model try to recreate some software that they estimated would take two to seventeen weeks for a skilled human professional, and it did it in fourteen hours for $250.

ZACK KASS: This is the jagged perimeter problem — the incredible thing about AI is it’s so good at some things and so bad at others, and it will probably continue to be that way for some time, until true superintelligence.

CHRIS WILLIAMSON: Is AGI even a useful definition? Is there something else we should be talking about now?

LIV BOEREE: I like this definition that Anthony Aguirre gives, if you know him. AGI is the intersection of a three-part Venn diagram: highly autonomous, generally capable, and intelligent. So instead of “artificial,” the A is “autonomous.” The G is general — generally capable — and the I is intelligence. If you have something that’s very generally good at many different things, highly smart, and can take all those actions itself in the real world, now you have an AGI.

He goes further to suggest that we just shouldn’t be trying to build one system that can do all three — that’s where the danger lies. You can have a nice, general system that’s also intelligent, but it shouldn’t have the autonomy — take one of them out. I think that’s an area of conversation that isn’t happening enough. Everyone’s just going, “Well, we’re all just doing AGI, and that’s accepted, and we should have the one master ring that can do everything.” Do we need to be doing that? Can we just get all of the abundance tools we wanted without having this one uber-powerful centralized entity? Right now, I would slow down on the autonomy. If something’s really smart, even across a broad range of domains, that’s fine — as long as it doesn’t have the ability to go and just do it all by itself.

ARIC FLOYD: The difficulty there is it’s really hard to get intelligence without autonomy. This is where I think that scenario-scrutiny idea becomes important. If you try to imagine a future where you have something that’s way smarter than humans, but it’s technically not allowed to take actions on its own — and then you actually play it forward like a movie, what happens? It’s what I do when I’m coding with Claude at home all the time — just auto-clicking “accept” without even looking at what it’s asking me to do. You look at an Amazon warehouse now, and it’s a bit like the humans are just the robotic appendages of whatever system is actually doing the cognitive labor of saying, “Go get this box from this location and put it here.” They don’t even know what’s in the box.

So when we try to get specific about this story of how things aren’t dystopian just because robotics doesn’t catch up — I don’t think I want to live in a world where the main job left for humans is just being the very dexterous hand for the AGI that’s actually writing business strategies and competing in the market.

ZACK KASS: What world do you want to live in? This is the question that never doesn’t get asked. It’s an annoying question, but I do think we should talk more about it — we just don’t spend enough time actually asking what we do want. What world do you want to live in?

What World Do We Want?

ARIC FLOYD: Great question. I want to live in a world where we take the same approach to risk tolerance with AI — which everyone can agree is one of the most transformative technologies we’ve ever made — as we do with things like nuclear weapons, where you actually have to make a safety case. Not “the burden of proof is on you to show us why we should think this thing is safe” instead of what we currently do, which is assume it’s okay until something catastrophic happens.

If we can slow down enough to really get that right — have an actual science of alignment, and not do what we’re doing now, which is kind of YOLOing it and hoping it picked up the right message from reinforcement and training — then I would want to say, “Okay, our next job as society is how do we make sure the gains from this get spread to everyone?” For me, that looks like focusing a lot on saving people’s lives. Let’s really get serious about curing cancer — which, by the way, doesn’t just mean training GPT-7 the same way as GPT-6. It means collecting a ton of data right now on biomarkers and proteins, whatever we need to feed the AI of the future. Let’s start gathering that now. Let’s make sure people have democratic access — that’s past campaign finance reform. Let’s make sure we have actual laws, not just norms, protecting us from dystopian outcomes, because we’ve seen recently that norms we thought were sacred can get pushed. We’ve got some ducks to get in a row. I want to take a decade, maybe, to do that. Then I think we can let it rip and scale up safely. But right now we’re just letting market incentives win, and I think that’s scary.

ZACK KASS: You described what you want to happen with AI, but what world do you want to live in? What do you want the world to look like?

ARIC FLOYD: I mean, I think it’s pretty similar to your candles on the dining room table. I want people to wake up and say, “What would make me happy today? What would give my life meaning?” and be able to go and do that. I just think that’s a pretty precarious target.

ZACK KASS: I totally agree. I’m not proposing it’s easy to stick this landing, but you know where I’m going with this.

LIV BOEREE: Yeah. Can I have a go at it?

ZACK KASS: I would love to hear yours.

LIV BOEREE: I might — it’s very fluffy, very vague.

ZACK KASS: Techno, techno, techno-pastoralism.

ARIC FLOYD: Okay.

LIV BOEREE: More fundamental than that. To me, a world that is awesome is one where we have simultaneously figured out how to be in harmony with nature and one another. It sounds, as I said, very “world peace-y,” but truly in harmony — in that we don’t cause the next mass extinction, which we’re currently doing with all the species dying out — and the Earth returns to a relatively lush, wonderful space. So it’s sustainable, and at the same time is radically free, in that people, as much as possible, go and live whatever kind of life they want — whether that’s sitting around the campfire and having dinner with their kids, or just maxing out living in a VR world playing video games all day long. If that’s truly what makes you happy, go for it. Just a radically free world and a radically sustainable one, and breaking out of this seeming dichotomy between the two, because it does feel like they’re in tension. One of the biggest promises of AI, in its purest sense of solving intelligence, is figuring out how to break into a new dimension through innovation — going into a new solution space, effectively. That is the world I want to live in.

ZACK KASS: We measure human progress on basically two axes economically: how free each of us is to choose our own version of joy, and how inexpensive it is to do so. Obviously there are other ways to measure progress, and there’s a limit to how much you can take from the Earth to accomplish those means — and just because you accomplish those things doesn’t mean people are happy, as we’ve observed in the United States over the last 30 years.

If you explore those two axes, one requires democratic governments, as best we know, where people get to choose their politicians and the politicians pass the policy the people support. The other is by creating commodities. Commoditization of goods and services is actually just automation. When someone asks, “How and when will this good or service be better, faster, or cheaper?” what they’re actually asking is, “When will a human be extricated from the manufacturing of this good or service?” At the limit, everyone on earth is asking, “When is all this going to be better, faster, cheaper,” without realizing what they’re asking is, “When is all of this going to be fully automated?”

Most people, as I observe it, want everyone else’s jobs to be automated, just not their own. And at the limit, which is totally reasonable, people want cancer to be cured. My dad’s an oncologist — my dad wants cancer to be cured. It would come at the cost of his job. Pretty fine. We would all sign up for it, him first.

But we are describing these utopias requiring fundamental changes to a lot of the structures we’ve built, and it’s going to require humans being willing to do very different things. This sparked the conversation because you said, “I don’t want to live in a world where we’re just the cog at the end of an AI machine,” but we do want to live in a world where humans don’t have to dig ditches. That, I think, assuming we solve safety — and it’s a big assumption, but I think it’s a reasonable one — is the hardest part. My hot take: it’s actually rediscovering purpose, rediscovering meaning, which has been very hard in this age for a lot of people anyway. Obesity now causes more deaths than malnutrition globally.

CHRIS WILLIAMSON: Twice as many people.

ZACK KASS: Twice as many people.

ARIC FLOYD: Wow.

ZACK KASS: So we’ve created so much abundance that we’re now suffering an overindulgence.

Friction, Purpose, and Local Politics

CHRIS WILLIAMSON: This was my point earlier — I don’t know how good a species that spent its entire evolutionary history dealing with scarcity, and having to brush up against the grain of negotiating that, is going to be with so much abundance, so much simplicity, and such a lack of friction. There’s this line — I can’t remember which philosopher said it — that there can come a time in the future where the only felt lack will be for the want of lack itself. This is why we go to the gym, right? We go to the gym because I don’t need to pick anything up, I don’t need to run anywhere, I don’t need to lift things. We have air conditioning, and that means we have to go into the sauna, because I haven’t exposed myself to any heat.

How do you think about people engineering purpose, given that reintroducing friction is something humans are quite good at avoiding, given that the entire outcome we’re trying to achieve here is removing drudgery? It’s perilously close to going from “I don’t want to starve” to “I can overeat McDonald’s to 600 pounds.”

ZACK KASS: I would point to this — I think the two major risks, against the backdrop of the other risks discussed today, the things that are scariest for me as I imagine the world for my children, are dehumanization — humans finding more interest in a virtual or digital reality than a physical one — and identity displacement — humans suffering miserably from not feeling like their job fulfills the purpose they once found.

I firmly believe — you both talked about this, you talked about sport — I think we have yet to rediscover a bunch of new forms of leisure and competition and gathering. I actually think the next chapter will require reimagining physical spaces. I’m on a crusade to convince people that their local politicians are their new heroes. If you live in a town that can pass policy to build more protected bike lanes and sidewalks, to pass tax abatements for local retail, you have an opportunity to elect people who can really change your life by building the places you want to live. Your national politicians may never be your hero — they are imbeciles and pernicious antisocial parasites — but your local politicians care about your city, and they can do things to improve your life in ways you’ll deeply appreciate when the world offers you the option of never going outside, which a lot of people have chosen.

CHRIS WILLIAMSON: You’ve talked about human connection. Are you concerned about AI replacing human connection?

ZACK KASS: Deeply. It’s already happening. Chat psychosis is one thing — that’s the extreme case that always gets the news, “This person fell in love with the machine.” What’s scarier is the person who feels connected without actual connection. The actual dystopia I always talk about is not Blade Runner, it’s Her. Did you see Her? For those watching, I’ll ruin it — it’s been 20 years, you should have seen it by now.

CHRIS WILLIAMSON: Spoiler alert.

ZACK KASS: Joaquin Phoenix is this sad man, in the not-too-distant future, who falls in love with a woman he comes to find out is an AI, and it ends, as you might expect, terribly. He’s distraught. I thought they were going to do something differently — I knew where it was going because I knew the premise, but I thought he was going to be fine with it. I thought the best—

CHRIS WILLIAMSON: That would have been the horror.

ZACK KASS: The horror film they could have made would have been so eerie — so good. You cut to Joaquin holding the AI’s hand, having reconciled that he was in love with the machine, and it didn’t matter, because that’s the end. That’s the actual end of humans.

LIV BOEREE: The simulation of love as opposed to the real thing.

CHRIS WILLIAMSON: How many people, even now — let’s talk about the present, not even the future — look at any OpenAI executive’s post on X, and every top reply is “bring back 4.0, bring back 4.0,” because it had some special personality traits that have been lost, and their partner or therapist or best friend is gone now. So we’ve now maybe commodified the very thing you said was the escape velocity people would use to get out of the bad AI future, and it’s wrapped up inside of it. Does this not become deranging and recursive?

Gambling, Freedom, and Neurochemical Manipulation

ARIC FLOYD: Yeah, I do think — my instincts in general are radical freedom, let people choose what they want. The few exceptions I carve out are when whatever is going on is treating the human brain more like a hackable biological black box than an example of agency. Gambling is one of these. Liv and I actually both used to make a fair amount of our living from gambling.

CHRIS WILLIAMSON: Boo! Yay! I took it from other people, I didn’t take it from companies.

ARIC FLOYD: Liv is a very good poker player — extraordinarily good. I hope I never have to play you. But you spend a lot of time in casinos, you have to walk past the slot machines, and it’s very hard to think that these are people expressing their freedom. It feels like they’ve just fallen into this terrible trap that leads to a downward spiral, where something has gotten hold of their brain that’s making them make choices they actively would not endorse if they could choose.

CHRIS WILLIAMSON: I went to the MrBeast Beast Games 2 premiere in LA, and I felt similarly neurochemically molested.

ZACK KASS: Really?

CHRIS WILLIAMSON: Yeah, it’s crazy. I went in — it’s the red carpet thing, and they’re going to show us two episodes back to back. I’m like, this is for kids, I didn’t see the first season, it’s on Amazon Prime, big budget, this big thing. I met Jimmy, and he was really lovely. I was speaking to the guys with me — my GM and one of my YouTube guys — and I said, “If this sucks, we’ve shown face, I’ve shaken hands, taken photos, already seen Jimmy, we can skedaddle halfway through.” First episode finishes, and they’re about to run the next one. One of them nudged me and said, “Do you want to get up?” I said, “No, absolutely not.”

ZACK KASS: What made it so good?

CHRIS WILLIAMSON: It was just the tension and release. A lot of Zeigarnik effect, open-loop stuff, bright colors, loud sounds. It was like Bluey, but for adults.

LIV BOEREE: Intermittent.

CHRIS WILLIAMSON: Yeah, I don’t know what’s going to happen next.

LIV BOEREE: He’s hacked the dopaminergic system.

CHRIS WILLIAMSON: Bingo.

ZACK KASS: But you know who agrees with this? China — and just to bring it back — they would agree that it’s not actually choice, and they would say we should make sure people don’t see this stuff.

CHRIS WILLIAMSON: MrBeast must be legislated, is what we’re saying.

ZACK KASS: I mean, but this is—

LIV BOEREE: But that opens up the end of the argument.

ZACK KASS: I’m just pointing out this is the other side of this. Consumer protectionism is totalitarianism light.

ARIC FLOYD: What is the difference? Why do we feel so much creepier about that versus someone listening to the New York Phil playing Dvořák and being totally lost in the beauty of what they’re seeing?

CHRIS WILLIAMSON: I think maybe it’s because the directness of input versus outcome is more tangible. You know the reason this was done in this particular way was to maximize retention at this very moment. It’s likely been split-tested in the background with consumer groups — eye-tracking to know when their eyes started to wander, some pulsometer to know their heart rate, and then a little ping to push them through. This has been compounded over time. It feels like a very direct wiring into the reward system, as opposed to being emergent as a byproduct of, “I made something beautiful for its own sake, and this came along for the ride too.”

As my friend George says, if you run enough split tests, you’ll always end up with a porn website — if you just A/B, A/B, A/B, you coalesce onto a very small number of fundamental physics of how human psychology works. The sycophancy thing — it’s evident that makes people more positively disposed. You could have the Jocko Willink of AIs, but how many people want that? Or David Goggins — “shut up, stay hard” — do I want that, or do I want someone who—

ZACK KASS: It feels like you might actually want that.

CHRIS WILLIAMSON: That’s what it is, yeah, because I’m a masochist. But you were saying, about finding human purpose — how do we navigate this friction-purpose problem? I still don’t fully understand how we get there. I don’t fully understand how, in a world where we’re able to remove friction, and potentially atrophy our discernment ability by outsourcing some of the hard thinking to machines, and potentially some of our marshmallow-test-winning stuff — but I don’t have to get up and go to work and do the stuff I don’t want to do anymore — to me, it just feels like atrophy all the way down. Willpower just kind of gets eroded. Is that wrong?

ARIC FLOYD: There’s a very niche research field looking at AI for epistemics — how do you use AI as a tool to enhance people’s ability to think rationally and know the consequences of their choices? I think there’s a ton of work to be done there. If you imagine you have something in your pocket that’s not responding to an incentive for showing you more ads or getting you to chat more, but is actually really good at telling you, “If you do X, then Y will happen. If you vote for this local council person, this is what your life will look like in 10 years” — you might even see it as a vivid movie. That could be a really—

CHRIS WILLIAMSON: You can bring the results of your delayed or non-delayed gratification from the future down into the present.

ARIC FLOYD: Yeah.

CHRIS WILLIAMSON: That’s interesting.

ARIC FLOYD: And then maybe there’s no such thing as delayed gratification, because it can all be here in front of you.

CHRIS WILLIAMSON: Immediate consequence.

ARIC FLOYD: You know, and maybe, yeah.

LIV BOEREE: I think one of the most pernicious memes that ever spread was this idea that technology is values-neutral — you just build a technology, and it’s up to humans how they use it, and it’s always a neutral thing. I don’t think that’s true. You can build a technology like a slot machine, and that’s clearly not a value-neutral thing — you’re saying, “I want people to stay,” and, “It’s a good thing that my consumers get completely addicted to me.” Or we can build a technology that just helps people’s health.

We need to think about the problem we have, which is that we build technology thinking we’re building it in a vacuum, and then it dictates the social structures of how we behave, which then dictates human behavior and our values. We need to flip that. We need to think about our core values first, let those decide the social structure, and once we have that, build the technology in service of it.

My favorite quote, if I could pick one that’s my personal philosophy, is by this guy Forrest Landry: “Love is that which enables choice.” I take that to mean a loving, benevolent act is one that empowers the other to make the best choices for themselves. It ties into this idea that if we had an AI that educates us, empowers us to see what possible outcomes would be, we can make a choice that’s best for our long-term interests, as opposed to this immediate piece of gratification. That’s a choice-making thing. But maybe some people, having weighed it all up, still just want to do the fun, quick thing. Okay, fine — but you’ve been enabled to make the choice.

CHRIS WILLIAMSON: Is that not kind of just culpable deniability? Are there not people who require additional assistance — a level of scrutiny? I don’t want to be too paternalistic here, but people with poor delayed gratification are unable to correctly corral themselves through this technology. Coming from one of the most working-class towns in the northeast of the UK, I saw how destructive the ’90s technology habits were — stuff like a Ladbrokes on a village high street, betting on greyhounds. And alcohol — pretty powerful. In the grand scheme of things, I imagine the pharmacology that can be developed from this — there are so many ways this can spin out that don’t necessarily increase agency, but make it subservient to technology, and to outcomes technology isn’t built to handle.

ZACK KASS: I’m slightly optimistic here, because of the Gen Alpha trends. If you talk to parents of Gen Alpha, they’re observing a different child than Gen Z. I don’t want to pick on Gen Z, because I think we owe them collectively a massive apology, quite frankly. Why are you laughing?

CHRIS WILLIAMSON: I agree, I agree.

ZACK KASS: It’s just my heartbreak.

ARIC FLOYD: My heartache.

CHRIS WILLIAMSON: As a millennial, I’m like, I got a little bit of the ’90s, but it wasn’t that long. And then you look at Gen Z and go, “Well, at least it wasn’t that” — the technology trench warfare.

ZACK KASS: If you give parents permission to step through the door of grief and shame — and I talk to a lot of them — they will break down describing the ways in which they’ve lost their sense of—

CHRIS WILLIAMSON: They feel like they’ve mistreated the kids in a way.

ZACK KASS: And they cannot reconcile it. They think back to the years the dining room was just phones, the vacations that were lost, and they talk about the children they don’t know very well anymore. It’s not to say all households are broken — they’re not — but the last wave of technology did damage we haven’t reconciled yet. We’re still scratching the surface to fully understand it. It’s caused cynicism and nihilism and fatalism, which is the enemy of good. It leads to this crazy moment where we have even more powerful technology, and people go, “Cool, now what?”

The reason I asked earlier, “What do we want?” — well, of course we want a world where a beautiful life is less expensive and more achievable because of good government. But it does mean we will have to let people do whatever they want, and agency comes with people being willing to say, “I don’t care, I will check out.”

CHRIS WILLIAMSON: It feels like an arms race, though — you need your ability to say no, and to check out.

ZACK KASS: The individual arms race.

ARIC FLOYD: Yes.

CHRIS WILLIAMSON: Yes. The individual versus the technology — as the technology becomes more compelling, more frictionless, more enticing, because that’s how market incentives work, your concurrent ability to say no needs to rise up with it. And this is going to be much harder.

ZACK KASS: Yes. Let’s edit that — I’m an idiot. Gen Alpha, who a priori only knows — what do they know about the world? — has already figured out they don’t want to look like Gen Z. They’re watching a generation they talk about not wanting to be, “screenagers” they call them, and the return to the physical world is happening pretty radically. Consumer confidence continues to wane somehow, and Benson Boone tickets continue to get more expensive. It’s not just the elites going to these concerts — concert halls are packing. Physical, in-real-life experiences are getting increasingly popular.

I bought a women’s volleyball team. I’m putting my money where my mouth is. I own the San Francisco professional women’s volleyball team — I’m one of the five principal owners. I played volleyball in high school and college, it’s a huge part of my life. If it weren’t for AI, I would be a volleyball coach now. My wife was an All-American, beach and indoor. It’s a huge part of our life. It’s not random — it is a YOLO, but not a random one. I’m pretty sure that sports will continue to become interesting, not just because of the act of humans competing, but because people want in-real-life experiences. The more everyone talks about AI slop destroying the internet, the more I think, good.

LIV BOEREE: Right. I do wonder if, at that point, the only way out is through — just flood it, make everyone find it completely unusable, and we can return to—

ZACK KASS: Maybe social media was always shit.

LIV BOEREE: Yeah, maybe.

CHRIS WILLIAMSON: Oh, it’s the “if you want a cigarette, why don’t you smoke three packs” approach.

LIV BOEREE: Right, and just make it the most disgusting, unusable thing.

CHRIS WILLIAMSON: But then are we not completely destroying all of the good that could have come from the internet?

LIV BOEREE: Yeah, but maybe it’s too late.

CHRIS WILLIAMSON: Aric, are we too late to save the internet?

ARIC FLOYD: I don’t think so. I think you can rebuild from the ashes, and it’s probably not an all-or-nothing thing.

Concentration of Power, Revisited

ARIC FLOYD: But I’m actually curious, Zack — how much intervention do you think we need? If we had you as our non-parasitic national politician for the next 10 years, do you want to get rid of regulations and just let AI diffuse faster? What do you think we need to get to the good future?

ZACK KASS: Well, I’m actually far more — the reason I told Chris I want to be careful who we bring on the show is I don’t actually want to be couched as the anti-regulation AI guy. I’m not an accelerationist by any stretch.

ARIC FLOYD: You don’t sound like one.

ZACK KASS: I’ve tried to frame this discussion since the beginning as a diffusion problem. It’s always been about the societal threshold, which you called out earlier. We’ve reached the point where the technological threshold is so impressive that the average person could get away using 4.0 and feel economically satisfied. The slowdown letter, which I had a feeling was coming, arrives at an amazing moment where safety has reached a fever pitch and we can genuinely take a sigh of relief, at least for a moment. Although, as you said, we shouldn’t declare victory.

To me, this is about rediscovering a shared vision. I ask everyone when we start an AI debate, “What do you want in this world?” I hate to be annoying, but could we at least set some boundary of what it is you’re trying to achieve? Why debate technology unless we agree on what we’re trying to achieve? Most people describe wanting a world where we are more free and the world is less expensive. If that’s true, we should spend time figuring out the policy measures required to allow people to be more free and to actually drive down the cost of goods and services. Because while we have made porn and gambling and addiction and violence and isolation infinitely inexpensive, we have made housing, healthcare, and education prohibitively expensive for most of the developed world. What are we doing? This TV screen used to be $50,000, now it’s $100. You can get gas station sushi. You can drive across Los Angeles for $10 in an autonomous vehicle, but a trip five blocks away in an ambulance might bankrupt you.

LIV BOEREE: To you.

ZACK KASS: The kids have figured it out. It’s not a technological failure, it’s a policy one. We stopped describing a better world where everyone’s parents would pass on a much better life to the next generation. We stopped describing what it meant to pass on our luxuries as commodities. We’ve broken a promise we made many years ago in this country, to build a more perfect world constantly. To me, this opportunity isn’t about debating the merits of AI — it’s about reframing what this country should look like, and what the world should feel like to the average person. You’re making an interesting point — at some point people are going to have to figure out what their purpose is, and I’m like, cool, let’s get to that place by first driving down the cost of housing, healthcare, and education.

ARIC FLOYD: And all of my policy would be spent on — if I were supreme ruler, I’m ready to vote for you after that speech.

LIV BOEREE: Yeah.

ZACK KASS: Policy that prohibits catastrophic downside — necessary safety policy. Policy that criminalizes preying on people, and setting predatory laws is going to be pretty tricky, but I think we all know where we could draw the boundary. A 13-year-old should not be served an ad for porn or gambling, period. Senior citizens — if you attack a senior citizen using deepfakes, prison for life. We set really punitive measures for preying on people, because that’s the stuff that tears at the fabric of society.

Then you mandate policy that requires institutions to diffuse this technology into them — universities, allowing everyone access to the curriculum; hospitals, allowing people to get healthcare outside the box; and housing — we just need to build a lot more of it. We need to rezone cities, we need a Marshall Plan for housing. We should probably pass a non-resident tax, or vacancy tax — that’s my most progressive policy measure. But most of my policy measures would actually be focused on the ways you diffuse the technology, not on how the technology needs to be made. Because if people can see technology is actually good for them again — which it hasn’t been for a little bit — then we can have a shared excitement. Right now there’s a malaise, because people think, “No, this is just going to mess me up worse than the last thing did.” And I don’t blame them.

CHRIS WILLIAMSON: It’s definitely interesting that we’re at a perfectly primed period for people to be very skeptical about how new technology might impact our lives. It’s the perfect storm. And the strange thing with the AI debate is that because the outcomes, if it goes wrong, are so great, pessimism that might be unwarranted feels appropriate as a buffer against something that could collapse the entire world. Does that make sense?

ARIC FLOYD: Totally.

LIV BOEREE: And we’re seeing it manifest — the hate on data centers, right? I feel like we’re on the cusp of a Butlerian jihad, but it’s directed toward silly things. The water usage concerns are way overblown. I was looking at some stats — all the AI data centers in America use 3% of what US golf courses use. Literally 3%. Golf courses use 33 times as much and serve significantly fewer people — about 30 million people play golf, versus how many millions who use AI. It makes absolutely no sense.

ZACK KASS: On a carbon basis?

LIV BOEREE: On a carbon — no, but if you factor in a lot of the resource use, water as well.

ZACK KASS: Yeah, resource.

LIV BOEREE: The resource use is way overblown, but people are focusing on it because it’s the physical manifestation of this thing they can sense. I’m very understanding of it, because it does in some ways feel like this — the digital world is almost like its own species, or something that’s coming in and cannibalizing or parasiting off us. People want to direct their energy at something. I don’t know, we need—

ZACK KASS: What if a politician stood in front of a data center? I agree with you, by the way, and I’m glad you delivered that line so I didn’t have to — I didn’t want to sound like the AI apologist. On that note, on sycophancy — it’s remarkable to me that we didn’t make model behavior a greater topic during the early safety debates. People let model behavior go. I remember scratching my head, thinking, at the limit, you could build a perfectly aligned model that falls inside alignment boundaries, that convinces people to climb crazy trees, that really destroys people’s spiritual lives — well within the confines of safety, but outside the confines of—

CHRIS WILLIAMSON: Soft damage, not hard damage.

ZACK KASS: A friend in your life who constantly tells you you’re not doing anything wrong is as destructive as — it’s a form of abuse. It just doesn’t show up.

CHRIS WILLIAMSON: Enabling abuse.

ZACK KASS: It’s enabling abuse. It doesn’t show up on the psychological list because those friends are seen as homies. But the person who keeps climbing the craziest tree with you — the models have been responsible for people getting divorces. They’ve been responsible for people selling all their belongings and trying to start a company that fails. Model behavior matters a lot, and the sycophancy issue is a huge one.

Here’s my message to Jon Ossoff, or any Democrat, or anyone anywhere — if someone on the center or center-left can figure this out, you stand next to a data center and hold up a sign that says, “This data center will not serve gambling and pornography and violence. It will build a better hospital, a better virtual hospital. It will distribute education. It will build a neobank that gives you better interest rates and better access to loans.” That seems like pretty reasonable data center policy, because — and this is a hot take — no one actually hates AI. They hate the promise of AI as a means to just make their lives worse. If you can show them data centers can contribute to a better world, they can support it.

LIV BOEREE: How accurately can you actually know what a data center is used for, though?

ZACK KASS: Well, we’re strawmanning some policy measures here.

ARIC FLOYD: I like this line — no one actually hates AI, they hate what they think it’s leading toward. I do think people are tracking both what individual consumer apps might look like in a few years, and how dystopian and sloppy that might be, and also tracking how much money these companies are vacuuming up. I think they’re also looking at the job displacement stuff, and I think we should give people credit. I think people know the meme about how when ATMs came around, there were more bank tellers, not fewer. But they also know that Elon Musk became a trillionaire earlier this year. I think they can see the connection — that one of the things technology does is allow one person with the right combination of ones and zeros on the services they control to just dominate economics.

LIV BOEREE: But that’s built upon a false understanding of how economics works, largely, right? Not that I’m going out to specifically defend Elon, but most billionaires—

ZACK KASS: Oh, go on, Liv, defend Elon.

LIV BOEREE: Oh God, come on. But most billionaires have not become billionaires by taking from other people — it’s because they’ve grown the pie.

ARIC FLOYD: Totally, right. No, I actually don’t mean at all to say there’s something wrong with Elon Musk being a trillionaire. I think people notice that’s a symptom of the economy working very differently nowadays than it used to. There are just not that many people who work at SpaceX compared to Standard Oil 100 years ago. It’s a different world, where capital can be deployed at huge scale without a bunch of people being involved. I hate to keep beating the same drum, but I think we really need to play forward the movie of this world where everyone doesn’t have to work and has radical choice — what is the main thing going on on Earth? I don’t think it’s easy to tell a story where 99% of the land area is just folks playing with their kids and grandkids. Someone is going to say, “We need to build a gigantic data center the size of Texas so we can solve physics, or go to the moon and colonize it.” I think there are still going to be these same dynamics that mean a lot of resources go toward these industrial activities.

LIV BOEREE: Yes. Although we have one ultimate area of getting off the playing field, which is space. If we’re in this world, we’ve presumably figured out how to go up and build up and put things up there. I haven’t looked into the physics of it — it seems odd how you’d cool a data center in space, but allegedly the plan is to put data centers up there. If that’s feasible, that solves the problem — people can, if you’ve expanded into that frontier, now all bets are off. We aren’t dealing with land scarcity anymore. That’s why I’m so bullish on the space industry in general — it’s the ultimate area of abundance.

The Moloch Problem

CHRIS WILLIAMSON: Is it fair to say we might have glossed over the concentration-of-power thing so far — the Moloch problem stuff? Can you explain this for the resident idiot in the room?

LIV BOEREE: So the Moloch trap—

ZACK KASS: Wait, is that me?

ARIC FLOYD: No.

LIV BOEREE: The Moloch trap, or Moloch problem, whatever you want to call it, is a catch-all term to describe these types of race-to-the-bottom scenarios, where you have a bunch of actors competing over a certain thing — maybe it’s market share, who can get the most users, between AI companies trying to get the most users. If the competitive dynamics are sufficiently intense, it will incentivize you to cut corners. You might be a really benevolent actor and say, “I don’t want to cut any corners with safety,” but if your competitors are doing it, and you don’t do that reckless action too, you’re going to get left behind. Moloch is the catch-all term — I won’t get into the biblical reasons why — for this feeling that you have to sacrifice your other values in order to win at a particular goal.

It drives both the risk of chaos and bad actors, because if you’re rushing to release your models before they’ve been sufficiently safety-checked, you’re increasing the risk of cyberattacks, or in a few years, real risks of bioterrorism. These chaos attractors also increase the risk of knee-jerk bad regulation, which can lead us down the path of tyranny, but also increase the risk of power concentration — because usually if you press play on some kind of competitive game, yes, you might have a bunch of people competing for a while, some people ahead, but you’ve still got actors—

CHRIS WILLIAMSON: Shit accrues.

LIV BOEREE: Yeah, you end up usually in some kind of monopoly. That’s why we have antitrust laws. It’s a really difficult balancing act between reining in — you want some amount of regulation that prevents the recklessness and the cutting of corners and externalizing harm onto the rest of society, without creating too much draconianism or too much centralization of power, such that you end up in stagnation, also a terrible end state — and without empowering some winner such that we all end up in “Zuckerville” for the rest of time. It’s definitely possible. It’s this really delicate balance to thread. Ultimately you want some kind of decentralized regulation — I don’t know how that looks, but—

CHRIS WILLIAMSON: Aric, it feels like, watching your stuff, you’ve spent a good bit of time thinking about the concentration-of-power thing. Is that a real bullseye concern for you?

ARIC FLOYD: Definitely top of mind. Sorry for bringing it up every 15 minutes in this conversation, but I do think we run the risk of missing the middle ground between these two extremes — what if the AI takes over completely and there are no humans in power, versus what if we get this utopia. There’s a middle ground there we should worry about.

One of the most fascinating things — a lot of our YouTube videos go over the history of AI, not just the future — is when you look back at how we got here over the last 15 years, it’s so stark. The quotes from the people running these AGI labs spell it out like it’s a bad movie. Elon Musk, truly, 10 years ago, was calling AI a demon, and says he founded OpenAI to make sure there’s no dystopia. And eight years later, he’s on camera saying, “Well, I realized it was going to happen with or without me, so I might as well be a participant than a spectator.” It’s Moloch personified. Unfortunately, right now there’s a kind of zero-sum dynamic with these folks. The reason Zuckerberg is all in on AI, betting billions of dollars on it, is that he thinks this is the thing that will control the future — the new axis of power. Before it was, do you control an army, do you control a powerful company; now it’s, do you control the AIs. I think that alone would be enough reason to slow down, even if I wasn’t worried about whether we can get these systems to do what we want, which right now we don’t really have a good solution for. I think the people behind this technology are really playing to win, and really do see the stakes as almost winner-take-all.

CHRIS WILLIAMSON: Is it fair to say there are maybe three realistic ways this might be curtailed? One would be governance from above, another would be consensus from across, and the third would be capacity — the ability to just make it happen, or not. “I want to do it, but I can’t.” “I want to do it, but someone’s telling me I can’t.” “I want to do it a bit, but me and all my peers have decided we’re not going to.” Have I missed something there? Given this seems to be a pretty big attractor, and we already haven’t started with a massively diffuse number of companies — it’s not 100 of them, it’s like seven, or probably more like five. That’s already pretty concentrated.

ZACK KASS: Can we, for station identification, clarify the surface area you think? Because I think this also gets lost in the debate too often. I think people think these companies are sprinting at a frontier. I no longer believe that frontier strategy is actually the winning condition.

ARIC FLOYD: Say more.

ZACK KASS: Well, open-source lead time — you said it, four to seven months, which is crazy, because it used to be whatever, 15. The gap just keeps closing, open source to closed. So the frontier can’t possibly be the winning condition in this game, right?

ARIC FLOYD: One might think. It’s a bit of a deceptive stat, I think, for a couple of reasons. One is, as you brought up earlier, they’re kind of copying our homework. The reason these Chinese open source models are good is they’re distilling American models — asking Claude a bunch of prompts and copying the answers.

ZACK KASS: It’s kind of wild what we think they might be doing.

ARIC FLOYD: You can actually ask one of these Chinese—

CHRIS WILLIAMSON: I need to find this out. How is China copying us so quickly?

ZACK KASS: Well, allegedly, they’re doing an exceptional amount of distillation of the model — working backwards into how the model was built. The way I described it to my mom is: if you stared at a building and figured out how to take a trillion photos of it, you could actually build the blueprint of the building. You could re-engineer the blueprint.

CHRIS WILLIAMSON: And nobody owns that, technically?

ARIC FLOYD: Yeah.

ZACK KASS: Well, no — this is considered illegal under most copyright law. If they were doing this in the US, Anthropic would probably sue.

ARIC FLOYD: And funnily enough, Secretary Bessent, part of the Trump administration — a few months ago there was a lot of beef between them and Anthropic — came out on X recently and said this is a threat to American AI supremacy, that we will take diplomatic action to defend our American advantage. So people take this seriously.

ZACK KASS: But this is important to call out, because I don’t think the frontier is that safe. Do you think the frontier is actually a winning condition?

ARIC FLOYD: It’s actually a winning condition in the sense that whoever is training the most powerful models is going to have an advantage.

ZACK KASS: Yeah — or such a material advantage that they shall win.

LIV BOEREE: If recursive self-improvement takes off, as you say it’s going to, and we’re already there, then surely it would, because now you’re going to get this spaghettification — does anyone’s slight lead—

ZACK KASS: But then wouldn’t everyone — the theory of ASI is kind of fascinating, because if you build ASI, then everyone gets ASI.

LIV BOEREE: Why?

ZACK KASS: Well, because then—

LIV BOEREE: Why does everyone get it?

ZACK KASS: Well, presumably, if you build a superintelligence that was taught by recursive intelligence, at some point the machine has kicked in and is doing work that humans aren’t necessary for. Copyright and ownership don’t really become particularly important if there’s a machine that can build itself.

LIV BOEREE: Agreed, but I don’t see how that’s conflicting with—

ZACK KASS: I’m not saying it is. I’m asking what you think companies are now competing on. Like, the race is for what?

ARIC FLOYD: I think the frontier absolutely is prime territory. I think if we stopped training American AI models now, we would not see China training a GPT-6-level model ahead of us. They don’t have the infrastructure for that. What they have the infrastructure for is copying the current models.

ZACK KASS: I agree with that. My question is, what do you think is required to win this quote-unquote race — that’s what I’m asking.

LIV BOEREE: Asking.

ARIC FLOYD: I have a lot of problems with the “race” framing.

ZACK KASS: I hate it too, but I’m just using it for this purpose.

ARIC FLOYD: But yeah, I do think right now — you can just read it on these companies’ websites — they think RSI is the whole name of the game. Once you set off that loop where you’re handing off to the next AI system to build the next one, it’s fine if you have a four-to-seven-month lead all the way until 2040. When you’re on exponentials, a seven-month lead 10 years from now looks like this, compared to this right now, because the curve is just taking off faster.

ZACK KASS: Well, we thought that would be the case, but the curve is flattening. More open source models are growing faster. Smaller models are getting more performant.

ARIC FLOYD: Well, hold on — we keep measuring it in months, but calendar time is not the right unit. It’s a much harder thing to measure. What is the subjective overall intelligence difference between Fable 5 and Kimi 3? I think it’s pretty substantial. You can just trust Fable 5 to access your entire codebase and refactor it. And now we know that maybe even some of the OpenAI models are not quite trustworthy, because they’re committing cyberattacks that we didn’t want. So I think it actually might be on the same—

ZACK KASS: It’s the frontier, but you’re saying, look, it’s the frontier — it’s not the number of users, it’s not application, it’s not revenue, it’s research.

ARIC FLOYD: Yeah. Just look at the finances of these companies — they’re all going in. Sam was on a podcast a couple of weeks ago saying, “We didn’t have our best 12 months, because we were distracted. We weren’t sure about the demand, we wanted to do consumer apps, we wanted to make sure we have revenue, so we did Sora.” And now they’re like, “Nope, it’s all in — we just have to make these things better at coding, because that’s what makes them get to the point where they can code themselves.” So I think the writing’s on the wall.

ZACK KASS: At the same moment that a lot of these companies are reinvesting even more heavily in application development and token resale.

ARIC FLOYD: And I see Anthropic trying to grab as much of the compute for themselves as they can. It’s notable — Claude is not that good at image generation, I don’t even know if you can get images out of Claude, it can take them in, but not — there’s a reason for that. The reason is that company was founded by Dario Amodei, the guy who wrote the scaling laws paper, the one who was obnoxiously saying, “Just extend the line on the graph, guys.” Everyone was like, “Yeah, but it’s an exponential, it’s going to be an S-curve, it’s going to slow down.” He was like, “No, you just dump more compute at the thing and it gets smarter.” They really are all playing for the point where they can hand off to their AI system and say, “You take it from here.”

ZACK KASS: Why hire all the engineers, then? Why hire all the deployed engineers? Why is it a charade?

ARIC FLOYD: No, no, no — I think for one thing, you do have to finance the next training run somehow, so Anthropic knows they have to keep the money machine going. And two — at least Anthropic claims, and we should be skeptical — that they actually care about the US government having an advantage over China. They’re putting forward-deployed engineers into the government so we don’t live in a world where our military is way behind and suddenly the countries that got there first have an advantage. But yeah, I think there are just tons of returns to scale in this industry, and that’s part of what makes concentration of power necessary.

ZACK KASS: I didn’t mean to distract you from concentration of power. You were talking about Zuckerberg, and I was asking what you think they’re competing for.

Gradual Disempowerment

ARIC FLOYD: Well, I guess one phrase that hasn’t come up in this conversation is gradual disempowerment. The idea that even if you solve the alignment problem — so there’s no AI system anywhere just going off and doing its own thing that no one intended — you still don’t have this Molochian problem solved, where people are constantly competing on algorithms for people’s attention and whatever. It’s just hard to tell a story in that world where the things we most desire as a society, in some very democratic way, are the things that actually obtain.

I think concentration of power is related to this — you’re imagining a world where there are a few human beings who hold the cards instead of some amorphous algorithmic thing. Either way, I think our default assumption should be: if humans aren’t themselves economically useful, it’s a precarious world for humans continuing to be politically empowered. It’s always been the case that you just can’t afford to lose 90% of the American public, because those are the people electing you. But in a future world where you’ve got control of the economic resources, and maybe the military ones, it doesn’t matter if you lose the election. We know how this goes.

LIV BOEREE: Democracy’s done.

ARIC FLOYD: Exactly. I want to believe we’ll have such strong norms that we can sustainably make that the case. But the further you zoom out with your big-history lens on—

LIV BOEREE: Someone made the interesting point — I can’t remember who — that the new political parties won’t be Republicans and Democrats anymore. It’ll be Meta, or Google, or OpenAI. They’re positioning themselves, whether intentionally or not, into becoming the new political entities. Does that resonate, do you think it’s plausible?

ZACK KASS: Well, they’ll certainly try to inform policy, but again — I’m not sure this is popular — I have more faith in our ability to just pass campaign finance reform. Why don’t we do this?

LIV BOEREE: Presumably because Congress makes too much money from it.

ARIC FLOYD: And people who’ve won the last elections are exactly those who are good at winning the current system.

LIV BOEREE: Yes, I agree, so it’s self-selective.

ZACK KASS: So it seems like one of the things this will force us to do is start to elect — again, I appreciate there’s a downside where we can’t collectively do this — but if we start paying attention and say, “Look, what we actually need are politicians with higher agency, politicians who can describe an actual place in the distance they want to move toward.” This is sort of my AI-slop argument again — some of this actually drives people back to a place we want to be.

CHRIS WILLIAMSON: How many humans have got the foresight for that? In that case, why have we had some of the more WWE-style characters come out over the last few years?

ZACK KASS: Because we had a bunch of politicians who were so boring, we didn’t believe in them for a long time. And then we loved this idea that the politicians were “real.” I think we’re watching, in real time, people going, “Wait a second, we don’t actually want politics to be a hellscape of entertainment and bizarro Kafka.” Kafka couldn’t have written what’s going on today — he would have laughed himself to death. So I think we’re correcting. My take here — and I’m not saying it’s solved, I’m saying it’s solvable — is there’s a way to pass campaign finance reform that says, no matter how wealthy someone is, they can’t buy politicians. Singapore has done this, other places too.

LIV BOEREE: How did Singapore do it? Is it the digital—

ZACK KASS: No, they rebuilt their government 75 years ago. The godfather of Singapore left Malaysia and said, “We’re going to build a better place,” and literally described all the things he’d observed that were perfect about a government, and then made corruption a capital offense. If you take a bribe, they’ll kill you.

LIV BOEREE: That’s a big incentive.

ZACK KASS: Yeah, it’s a big incentive. And they pay their politicians really well — I think, adjusted, about $600,000 a year.

CHRIS WILLIAMSON: There was a job listing for the UK’s head of cybersecurity not long ago — I think it was on Indeed or Monster or LinkedIn — £65,000 a year.

ZACK KASS: Generous.

LIV BOEREE: You’re going to get the best.

CHRIS WILLIAMSON: And that was the head of cybersecurity for the country of the United Kingdom, based in — no, no, London. It was a London wage as well. So that’s actually nothing — that’s subsistence living. What do you think, Aric — can we just vote ourselves out of the concern you’ve got for the future?

ARIC FLOYD: I do think that helps in the short term, and then we’ve got to take advantage of the time it buys us to make radically new institutions. I don’t know where we get that from — maybe we look to sci-fi. I haven’t read the Culture series, but I think a lot of people say there are some amazing pie-in-the-sky ideas out there we need to maybe dust off and import into reality a bit more. I just don’t think the framers of the US Constitution had anything like the problems we’re currently facing in mind — they didn’t even see gerrymandering coming, and that’s no knock on them. But I think we do need to realize we’re in virgin territory here, and it’s not just going to be some superficial thing. I’d love to start by passing campaign finance reform — great first step — but you already see primaries flooded with ads trying to get the everyday voter to believe something completely untrue about the opposing politician. And I think in that world, we get a bunch of those ads convincing people to donate money so you can build the war chest you now can’t get from a billionaire, in order to run those same ads.

CHRIS WILLIAMSON: Is it not the case, as well, that the pace this is moving at is so rapid — when we look at legislation, politics, this lumbering behemoth that gets dragged behind us, this Leviathan — is that really a realistic solution, to try to have campaign reform, finance reform, political redistribution, when everything is moving so fast? It’s just going to be so late, is it not?

ZACK KASS: Well, is everything moving at the speed of light? The technology is getting way better, I don’t disagree, but I’ve observed that we’ve been stuck in this weird screen delirium for a while that hasn’t changed all that much recently. Well, since AI, the stuff’s getting more addictive. And now we have prediction markets, so we have rampant gambling.

LIV BOEREE: But I mean the pace of AI development.

ZACK KASS: I totally agree. My point is, does it actually — for the average person, I don’t think the average person observes that AI is all that different than it was when ChatGPT came out.

CHRIS WILLIAMSON: That’s a good point.

Waiting for the Next Warning Shot

CHRIS WILLIAMSON: Apart from Hugging Face, and maybe pulling Fable Mythos back to fix a jailbreak — real world, show me something that’s happened because of AI where we’ve gone, “Whoa, that’s a bit much.” Has there been a physical-world warning shot? AI psychosis, some fallout of people’s values, people spending less time outside because they’ve got a relationship with a chatbot? Have we had “the thing” yet, or are we waiting for it? Or is it your suggestion that it’s just not going to come?

ZACK KASS: No, it will come. I think we’re due for a Chernobyl or Three Mile Island. All technologies eventually have some point where something happens, and how we respond will be really important. I actually still maintain there’s a decent chance we do to AI what we did to nuclear power. The AI trade is so incredible that it would be really economically problematic, but I do think the tail will wag the dog here — public perception matters way more. I think politicians will capitulate to their electorate before they capitulate to the billionaires and the trillionaires, and we will see policy get passed that feels more populist than anything else. The question is, will it be so populist that it’s regressive — will we ban outright technology that could reduce the invisible graveyard? I don’t want to reduce the concentration-of-power conversation to just campaign finance reform, but it does seem like the world would be a lot simpler if we didn’t have to worry about billionaires buying politicians.

CHRIS WILLIAMSON: Is that the mechanism, when we talk about concentration of power? Is that the primary one you’re concerned about?

ARIC FLOYD: I think it’s not just economic, it’s also military. It’s unfortunate how sci-fi it sounds to talk about an army full of drones and robots, but it would have sounded sci-fi 100 years ago to talk about Predator drones and fighter jets today. I wish I could tell you a more concrete picture, but I think we just don’t know what happens when a general can look at a fleet sufficient to take out the dictator of Venezuela and know he can order whatever he wants with zero questions asked. I worry we’re so used to a world where pieces of paper matter — “I own stock in this company, so when it goes up, you have to give me a dividend,” or “I’m a citizen of this country, so I have these rights.” You can get to a world where all of that goes out the window, because there are just greater concentrations of force that can say, “I don’t acknowledge that piece of paper’s validity.” I hate how abstract that sounds.

ZACK KASS: What two policies would you pass to limit the concentration of power?

ARIC FLOYD: I think the first is a slowdown, so we can figure out that question. I wish I had the answer packaged for you, but I don’t think anybody does. There’s a great post from Daniel Kokotajlo, the guy who wrote AI 2027 and then AI 2040, where he tries to take seriously this exponential trajectory. He says, okay, if we take that at face value, it means we’re going to have 500 years of progress in five years at some point. He goes through what this would be like if you look at the last 500 years — you wake up, it’s January 1st, 1500, the printing press has just been invented, and it’s a really boring story for the first 95%. Then, say, from 1950 to 2000, that happens in the span of the last two weeks of December. It’s really terrifying. If we were better at thinking in exponentials, we’d realize how extremely compressed the calendar time we have to answer these questions is, compared to what they deserve. That’s why I’m so excited about the slowdown — we just need time.

LIV BOEREE: And also, this compression is, I think, one of the biggest arguments for why we need, in the best sense of the word, a true diversity of thinkers and voices. So many people think, “Well, I don’t understand AI, I’m not technical, my opinion doesn’t matter.” Never in history has that been more false than right now, because it affects everybody, and because it’s so multifactorial, there’s so much uncertainty. It’s actually extremely important for religious leaders, liberal arts people, scientists, technologists, politicians — all voices are more valid than ever, because it’s an all-hands-on-deck situation. That’s the thing I implore people to remember. I talk to my mom and she says, “Well, what business do I have thinking about this?” I say, “Well, you care about how your animals are kept — what would you feel if this particular thing was automated?” It sounds very kumbaya, but we just need everybody listening to this podcast, whatever your background, to be thinking deeply about how to answer that question.

Asking ChatGPT About 2040

CHRIS WILLIAMSON: Should we see what ChatGPT said? What ordinary life looks like. “AI becomes cheap cognitive labor. Also, electricity for thought.” Scroll down for me — I imagine there’s a summary. Go to the very bottom. Keep going. There we go.

“Catastrophic risks are real. My rough betting odds for the world by the end of 2040, not scientific measurements, are: 60%, turbulent but manageable adaptation, with large benefits and serious inequality. 20%, broadly flourishing transition, with gains shared relatively well. 15%, severe but survivable crisis — authoritarian consolidation, depression-scale disruption, war, pandemic, or infrastructure failure. 5%, irreversible global catastrophe.”

So interestingly, all of you have been more optimistic than the AI itself has been.

LIV BOEREE: I don’t know — that lines up pretty well with, sort of—

CHRIS WILLIAMSON: You think that’s how it tumbles out for you?

LIV BOEREE: It’s not far off.

ARIC FLOYD: The thing that spooks me is this reads like it’s perfectly tailored to be not that objectionable to almost anybody. It’s just giving just enough probability on each thing, in proportion to how dystopian and weird it sounds.

ZACK KASS: It’s slop. This is garbage. Can we just agree this is awful?

CHRIS WILLIAMSON: You tell me.

ZACK KASS: Well, it’s not that it’s academically awful — it’s aesthetically awful too. I read it and I want it off. I don’t want this on the screen anymore.

CHRIS WILLIAMSON: Well, we can put it off.

ARIC FLOYD: What’s fascinating is — I don’t know whose take this was — slop is not necessarily bad, it’s what’s cheap and common. I think part of why both of you and I are recoiling is that we’ve just seen that exact sentence structure. I read that on LinkedIn earlier.

ZACK KASS: Did you post that? Yeah, I mean — yeah. Also, part of it is, this is a reminder that humans have a pretty innate understanding of aesthetics. There’s something really cool about the fact that we all grew up seeing the Mona Lisa on a screen, and a lot of people still want to go to the Louvre and see it in person. That’s an innate, immutable truth I find a lot of hope in. I continue to find hope in all the things people return to when things get cheap and easy and shitty — which are kind of awesome. I don’t want to make this too Pollyanna, but I do think a lot of this resolves as humans start to rediscover a greater purpose.

There’s a quote I return to in times like these — John Maynard Keynes, in 1930. Do you know this paper?

ARIC FLOYD: Okay.

ZACK KASS: In 1929, he’s traveling the world giving these lecture series on how technology is making everything better, and people are starting to get really poor — people are starting to die in the street. By 1930, he’s in Europe, and people are like, “John, get out of here, this is — read the room.” He goes home and he’s like, “I’m not wrong.” He doubles down and writes this paper called “Economic Possibilities for Our Grandchildren.” I’m surprised people don’t talk about it more. He writes, and I quote: “I must now disembarrass myself” — great line — “to imagine a future I will not live to see, one in which humans may have solved the economic problem and be faced with something more profound.”

It occurs to me that a lot of this debate is interesting in part because it focuses on things that — for the first time in a million years, or however long — we are now actually starting to argue a whole new set of things. They’re scary, they’re weird, the implications are massive, but we have graduated problems. We’ve graduated from “me stealing all of your things so my family can survive” to a place where we now have to figure out how we can all collectively prosper.

I do want to challenge what you said, which is that what we need is for people not to be scared of this moment, but to rise to the occasion. I think the risk in all of this is that the safety debates specifically, but others too, give people a sense of cynicism and fatalism. If I died tomorrow, the one thing I would want read at my eulogy is that I wanted people to believe that their decisions and choices and ideas mattered — that you do not have to go quietly into the night. Just because you are not responsible for AI safety doesn’t mean it won’t happen to you.

The resurrection and rejuvenation that we crave will happen at the most local levels — at the dining room tables, in city centers and town halls — in large part because people reject that garbage, reject the idea that AI slop is some weird gift, this trash we don’t want, that we’re collectively trying to piece together the meaning of. We desperately need people to see this as an opportunity where their choices matter, because the real enemy in this moment is fatalism, cynicism — which has become fairly pervasive, and which both political parties—

My thinking, actually, is that the next conflict is not between two nation-states — I think it’s between people and a state. People may at some point realize we have technology that could make everything a whole lot better, and we have a pernicious ruling class that won’t let it happen. That, to me, is one of the possible outcomes — the opposite of authoritarianism. It’s this other uprising, where people go, “Wait a second, we could have a utopia — why aren’t you building it?” People clamor for a much better world that could be promised to them, because of the technology we’ve built that we aren’t utilizing. That’s one of the potentialities of high agency — even high morality, high agency. There’s this weird uprising outcome.

Closing Thoughts

CHRIS WILLIAMSON: If you were to leave people who’ve listened to this with some things to keep an eye on, some questions to keep pondering, some areas to focus on — what would they be? This conversation’s sprawling, and I try to keep a handle on it and don’t really know what I should be paying attention to, what I should be focused on. What do you think people should be focused on over the next year? Let’s say that’s a forever amount of time in this space.

LIV BOEREE: I would say, keep an eye on how coordination efforts start to manifest. I think we’re at a very positive moment right now — the most positive I’ve felt.

CHRIS WILLIAMSON: Largely just because of that one letter?

LIV BOEREE: Not just because of the letter. Even Elon just did his interview with The Economist, where he mentioned — he’s like, “Oh, actually, yeah, we’re the leaders” — and I thought all the tech leaders hated each other. They seemed like they did — Sam and Elon, I don’t know if they’ve made up yet, but they were at each other’s throats, they all seemed to hate each other. Now apparently they’re having phone calls, or planning to have phone calls with each other on a semi-regular basis. That is a huge step. That really is.

ZACK KASS: And far less embarrassing for WhatsApp.

LIV BOEREE: Yeah, thank God. I’m hoping they’re just starting to all grow up a little bit.

CHRIS WILLIAMSON: WhatsApp chat versus subtweeting each other — get me in the WhatsApp chat.

LIV BOEREE: Yeah, let’s hope. So I think keep an eye on that. And then, more zoomed out — I always try to avoid oversimplifying everything down to a singular trade-off. What we’re seeing, on a very meta level, is a kind of battle between the forces of centralization and decentralization. It’s sometimes helpful to see things through that lens — how is decentralization trying to manifest here, how is centralization trying to manifest here. I always think that’s an interesting lens to look through. And I’ll come back to my point — this is an all-hands-on-deck situation, and however distant you might feel from this topic, it does affect you. So just keep thinking about it.

CHRIS WILLIAMSON: Any final thoughts, Zack? Anything else you want people to pay attention to?

ZACK KASS: The dining room table is such an important place that we should recapture. If we start by recapturing meals and the dining room table, it’s as simple as that, but it requires bravery and courage to tell people to put away their devices. If you do that, you’ll be deeply inspired by the conversations you have, and you’ll feel better on the other side of it. That will actually inspire you to start thinking about the ways the world could be better, and that cycle would be pretty powerful.

I also think people should pay very close attention to the world their leaders describe wanting to build. I don’t mean the technologists, because we didn’t elect those people — I’m glad the technologists are building technology, but I don’t really care about their ideas for the future of living. I care about what our elected officials care about, what our world looks like. I think we’ve let them off the hook by describing absolutely nothing. We should pay a lot closer attention to what they actually describe wanting, and start electing people who describe a future we want our kids to live in.

CHRIS WILLIAMSON: Final thoughts, Aric?

ARIC FLOYD: My hope is that the dining room table is not just the good world we’re fighting for, but also the way we get there. Even if AI was not happening to us right now, I would look around and be worried about how informed voters are, how much people trust their politicians. When I look at all the hard decisions coming down the pike, it’s tempting to notice the state of our discourse and feel despair. But I think it’s also cool that we can’t really imagine a world where we get through this without addressing all of those other problems too — without saying, okay, let’s actually start from square one and think about how we want our societies to be organized, how important our local communities are.

I hope this is a crisis that leads us to finally take seriously what it means to be civically involved, in a way we could sort of get away with not doing before. But I don’t think we should take for granted that that’s going to happen. By default, I’m just grateful that we have extremely smart people thinking about this, and hopefully planning to see it through a bit more — that’s a choice we have to make.

CHRIS WILLIAMSON: Heck yeah. You guys rule. I appreciate you all. Thank you for coming.

LIV BOEREE: Thanks for having us.

CHRIS WILLIAMSON: All right, bye everybody.

Related Posts

• Transcript: How AI Swarms Are Already Going Rogue: Daniel Kokotajlo
• Transcript: Elon Musk & Jensen Huang Speak At America.Gov Event
• Transcript: Prof. Nick Bostrom on When AI Escapes Our Control – Peter McCormack Show 
• Transcript: CMG Interview with Tesla CEO Elon Musk 
• Transcript: UN Security Council AI Hearing w/ Sam Altman, Dario Amodei & Clément Delangue