⏩ FP! Week In Review #41

Also: Key Up Wing and Down Wing news from the week that was

Jul 18, 2026

In Case You Missed It ...

✨ My interview with Sebastian Mallaby, author of ‘The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence’ (Tuesday, 35 minutes)

Please check out the full transcript of this marvelous chat, below at the end of the issue. Perhaps my favorite exchange is when I ask Mallaby why so many economists are skeptical that AI will become this utterly transformative technology. His response:

I think the frontier of technology is accelerating really, really fast. If you look back to 2022 with the arrival of ChatGPT, and then you just think what’s happened since then, you go from a model that hallucinated nonstop, to a model that mostly stopped hallucinating when you plugged in GPT-4 a few months later, to a model with a context window where you could upload the entirety of a Tolstoy novel and query it about the novel. Then you get reasoning, so that logic and mathematics becomes possible. Then you get agentic models. You get these coding assistants. They can handle multimodal inputs and sound and video. The number of improvements, before we even talk about Mythos and the cyber capabilities of that one, I mean, it’s been an extraordinary ride in what is actually less than four years. So, I don’t take back anything I say about the speed of the advance of the frontier.

So, I don’t agree with him, but I think I would put the mainstream of the economics profession between him and the assumptions you get from the pure technologists in Silicon Valley. In other words, the rollout is going to take a while. It’s going to take a while for corporations to figure out how to make this productive for them. There are certain applications, like writing code, where we are already there. It’s already super helpful. It’s beginning in things like law and call centers and so on. But if you want to reengineer fundamentally how your business is structured and close down departments and all that, we’re not there yet. So that’s going to take a while. It’s instructive to remember how many years ago people said, “Well, there’ll be driverless taxis absolutely everywhere.”

The Infinity Machine by Sebastian Mallaby: 9780593831847 | PenguinRandomHouse.com: Books

✨ Superintelligence soon: How seriously should I take the AI 2040 scenario? (Wednesday, 1333 words)

There’s this legal doctrine about witness testimony: falsus in uno, falsus in omnibus—false in one thing, false in everything. AI scenario-building may need a variant: absurdum in uno, absurdum in omnibus. If supposedly serious speculation about the Age of Artificial Intelligence depends on an absurd premise, the whole magilla might reasonably be treated as absurd.

That seems to be where I’m at with AI 2040, the buzzy continuation of AI 2027. Its preferred Plan A—an international US-China deal delaying superintelligence until 2040—is basically back to “The Pause” vibe of early 2023. To be sure, my problem isn’t so much the technological mechanism. As readers know, I take AGI and superintelligence as AI advances that could happen, and sooner rather than later. As such, we should be thinking hard right now about the potential socioeconomic and national security impacts.

My trouble is the underlying economics of AI 2040—especially the notion of a “controlled explosive growth” scenario of 50 percent real GDP growth in 2032, averaging 100 percent through 2037, with employment plunging to 12 percent. If too much creative destruction is possible, there you go. This doesn’t work on so many levels, including the certainty of intense public backlash. (We live in a world where Rhode Island just regulated self-checkouts to guarantee jobs.) If superintelligence implies warp-speed growth, yet economics and political economy suggest such growth is impossible due to real-world, non-technological constraints, something in the story here doesn’t work as presented. Absurdum in uno, absurdum in omnibus?

✨ Down with American data centers! Up with Chinese AI! (Friday, 1031 words)

There might be a way the Sputnik-flavored Axios headline “China just erased America’s AI lead” is true. Kimi K3, a new open-weight model from Chinese startup Moonshot AI, has topped a popular coding benchmark. Plus, it’s cheaper and soon freely downloadable. Yet the best American models still lead on broader measures. So maybe Kimi is at about the level of Anthropic’s Opus 4.8—more or less, but likely more—which is still pretty impressive since that was the company’s flagship model until Fable 5 arrived last month. I think that gives a broad sense of the current US lead at the frontier.

Now some further context: New York state just imposed a year-long moratorium on new hyperscaler data centers—at least very big ones. Activists whose dream is “no data centers at all” wanted even more. As JPMorgan warns, “The data center backlash is real—and becoming financially material.” If the economic case stays strong, data centers will get built, but more slowly and at greater cost. I’m sure that would be to the great delight of both Chinese AI firms and the Chinese Communist Party. And the case for orbital data centers becomes even stronger—which is kind of absurd, but Wall Street analysts are already running the numbers.

The Down Wingers have held an obscenely strong position in our society for a half-century. What we are seeing right now is evidence that they will not easily be forced into permanent retreat.

On sale everywhere: The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

⤴️⤵️ Up Wing/Down Wing

A selection of pro-progress and anti-progress news items from the past week.

⤴ Up Wing Things

AI & Economy

AI isn’t destroying entry-level jobs. It’s changing them - FT
Tyler Cowen: The Future Belongs to AI Maniacs - The Free Press
U.S. Workers Are More Productive Than Ever. A.I. Isn’t the Key. - NYT
Why AI Might Actually Help Solve the Next Labor Crisis - WSJ
We need humanities more than ever in the age of AI - FT
Socialists think wealth is stolen. They’re wrong. - The Washington Post

AI Industry

TSMC Plans Another $100 Billion U.S. Investment as AI Demand Surges - The Information
ASML raises forecasts as AI boom drives chipmaking demand - FT
Mira Murati’s AI Startup Releases First Model in Bid to Loosen AI Giants’ Grip - WSJ
Uber and Waymo Are Sparring. The Robotaxi Future Has Arrived - Bloomberg
AI Isn’t Smarter Than a Baby—Yet - WIRED
AI’s black box problem: discovering physics we don’t understand - Physics World

Energy & Infrastructure

Space

We’ve found a rocky, temperate planet’s atmosphere for the first time - New Scientist
The whole world looking up: inspiration from the Moon - The Space Review
Payloads used to dictate the terms of launch. That’s finally changing. - Ars Technica
China recovered its first reusable rocket and showed a new way to do it - Ars Technica

Health & Science

Best treatment for multiple sclerosis may be antivirals - New Scientist
Game that reduces dementia risk may clear amyloid from men’s brains - New Scientist
Five Exciting Findings From Recent Anti-Aging Research - Real Clear Science
We’re finally learning what happens to gifted children in adulthood - New Scientist

Frontier Tech

PsiQuantum has a plan to make a massive quantum computer out of light - MIT Technology Review
The Return to Flying Faster Than Sound Will Start Small - Bloomberg

⤵ Down Wing Things

Data Centers

New York Governor Signs First Statewide Data Center Moratorium - WIRED
New York becomes first state to impose data center moratorium - Washington Post
Will New York’s Data Center Moratorium Actually Stop Anything? - Heatmap
‘I wouldn’t call it panic’: Industry quails at Hochul’s data center pause - POLITICO
New York Governor Kathy Hochul Is Walking a Narrow Lane on Data Centers - Heatmap
New York’s Ban on the Future - The Free Press
Is This the Fastest Opinion Shift in American Politics? - NYT
AI Data-Center Construction Is Booming—but Not Much Else Is - WSJ
Data Centers Are Quietly Taking Over Texas. The Pollution Could Be Catastrophic - WIRED
Australia Plans to Govern Use of Water, Power for AI - WSJ

China & AI

China just erased America’s AI lead - Axios
China’s Moonshot AI Unveils Kimi Model, Threatening America’s Lead - NYT
How China Is Taking Control of the Future of AI - Bloomberg

AI Backlash

The AI Backlash Has Tech Executives Fearing for Their Lives - WSJ
The Hard-Line Activists Ramping Up for the War With AI - WSJ
AI is the Democratic Party’s Next Villain - Free Systems Substack
Nobel economists, tech leaders warn how AI could threaten jobs - The Washington Post
Meta Workers Accuse It of Using AI to Conduct Discriminatory Layoffs - WSJ
Meta Removes A.I. Feature on Instagram After Days of Backlash - NYT
The looting of science fiction - Aeon

Science Policy

Health & Science

Economy & Business

↕️ Which Wing Things?

China & AI

China Economy

AI & Society

How AI Is Rewriting Human Nature - Noema
Are you there, Claude? It’s me, Margaret. - The Washington Post
Why We Demand Perfect Machines Yet Tolerate Human Carnage - Noema
One of sci-fi’s most difficult questions about AI is becoming real - The Washington Post
AI Isn’t Human. Stop Talking About It Like It Is. - The Free Press
Jonathan Haidt is wrong about age-gating the internet - The Argument

AI Safety & Governance

The Most Important Words in the Battle Over AI - POLITICO
Demis Hassabis has a plan to harness AI safely - The Economist
Hassabis’ AI Standards Idea Gets Support—What’s Next? - The Information
He warned AI could lead to extinction. Now he says there’s a better path. - The Washington Post
AI Researchers Are Having an Identity Crisis - The Information

AI & Economy

Tyler Cowen: The Future Belongs to AI Maniacs - The Free Press
AI’s Next Big Mission Is Rewiring Your Workplace - Bloomberg
AI Data-Center Construction Is Booming—but Not Much Else Is - WSJ
Stubborn Inflation, Better Growth: Economists Weigh In - WSJ
Is the US or Europe better off? - FT
DARPA Wants to Build an Army of 100,000 AI Agents That Can Think and Act on Their Own - The Debrief

Space & Frontier Tech

SpaceX’s Blow-It-Up Testing Won’t Fly on Starship - Bloomberg
How hard is it to build orbital data centers, actually? - Ars Technica
Simulating everything, sort of: The promise and limits of world models - Ars Technica

My chat (+transcript) with author of The Infinity Machine, Sebastian Mallaby

(A lightly edited transcript of our podcast conversation)

In This Episode:

● The Quest for “Success” (0:27)

● Inside the Mind of Hassabis (8:29)

● The Race for Monopoly (12:31)

● The Economics of AI Anxiety (17:05)

● Governing the AI Race (24:17)

● The Biggest Leap Since Abstract Thought (30:13)

The Quest for “Success” (0:27)

You look backwards; you see how fast the progress has been. To merely extrapolate forwards is probably to undersell the speed at which we’ll accelerate in the future because there’s an accelerating phenomenon here, where the more advanced you are, the easier it is to get to the next level.

Pethokoukis: Sebastian, right there in the introduction of the book, sentence number three, you write, “At some point in the not-so-distant future, artificial intelligence will beat human intelligence at almost every mental task. And to say this marks a watershed would be a parody of understatement. AI holds a transformation more profound than anything since homo sapiens acquired the capacity for abstract thought some 70,000 years ago.” In my view, you sound like a true believer that this quest will be successful. Was that your belief when you began the book?

Mallaby: Yeah, I mean, I began the book in 2022 and you already had a couple of real breakthroughs. One was the AlphaGo system that DeepMind had built, which surpassed all the best human players. That was significant because obviously Go is this massive combinatorial space. There are 19 by 19 squares on the board. So the very first move, there’s 361 options. The second move, there’s 360. You multiply that out, you get to a number pretty close to infinity in terms of the number of permutations that you could have.

Yet, the machine figures out the patterns and figures out the best strategies and can master that infinity of data. Then the same thing, but actually even more extreme, with protein folding in 2020, another DeepMind system, DeepMind being the startup that Demis Hassabis, my main character, had started in London in 2010. Again, protein folding, just an enormous number of possible permutations and finding the way that a protein is going to fold itself up like a self-executing origami model is like searching for a needle in 10 to the 300th haystacks.

I mean it’s just this incredibly big number of possibilities. So, you had two instances where infinity had been conquered, hence my title, The Infinity Machine. It felt in 2022 when I began the project that this was going to break out. I didn’t know it would break out quite as fast as it did. In fact, I pitched Demis Hassabis on the idea of this book in November of 2022. At the end of that month is when ChatGPT came out. So, I was lucky that it broke out immediately into the mainstream after I began.

I just think you look backwards, you see how fast the progress has been. To merely extrapolate forwards is probably to undersell the speed at which we’ll accelerate in the future because there’s an accelerating phenomenon here, where the more advanced you are, the more easy it is to get to the next level.

I hope you don’t object to my use of the phrase “true believer,” with its quasi-religious aspect, but you have confidence this quest will be successful. Are you enthusiastic about that? There are people who believe it’ll be successful who are not enthusiastic about that, whether for all manner of reasons, ranging from they don’t like data centers to they think the robots will take all the jobs and, when they’re done taking all the jobs, they’ll kill us.

Well, that’s a more complicated question, and I do have mixed feelings about that. I mean, I think, in one sense, you can say obviously there are huge upsides and huge downsides, and anybody honest is going to recognize both, but then also perhaps reckon that humans have always faced this trade-off when confronting technological disruption. They go forward because that’s what defines being human. The willingness to accept the uncertainty, the thrill, the adventure of a new technology coming down the pike. I mean, we just never stop it. We don’t choose not to go forward.

So, presumably, we’re going to navigate this one and we’ll be okay, and we’ll manage the downside somehow. But on another level, one could debate the optimal speed of this transition. I mean, maybe society would have a better shot at digesting this if it could be stretched out. If I project that, let’s say we take the Anthropic projection, which is recursive self-improvement (i.e. the machine codes the next version of the machine), this is projected to arrive at Anthropic at the end of 2028.

That’s their base case. Let’s say they’re a bit optimistic, let’s say it’s a couple of years after that, but we are very close now to a situation where the frontier model codes the next frontier model. Now, would I prefer that to be ten years in the future, not three or four? Probably. I think it would make it easier for society to digest it.

That reminds me of a conversation I had with the economist David Autor, who’s written about the China trade shock. I recall asking him, given the disruption, would it have been better somehow, as unlikely as it had been to not do that, to not open with China, though obviously I think that would be impossible? He’s like, “I guess the only thing I would say is perhaps if we could have slowed down somehow our exposure, and exposing workers to this humongous other labor market, and given people more time to adjust, perhaps that would’ve been a good idea.”

Again, I don’t know how likely that is with AI. Having spoken once myself to Demis Hassabis, he’s a very calming presence. He does not seem wild-eyed. This was maybe a year ago. He sounded like a person who picked his words with great care.

Given that, in your many conversations with him, did your feeling about the wisdom of this project change at all during the writing of the book? Was there ever a point where you were like, “I’m not sure this is a good idea given all the risks,” but talking to him made you think, “Yes, one, it’s inevitable and, two, it’ll be okay. We’ll figure it out, and forward we will go”?

I would say, actually, I became a bit more pessimistic as my project moved along, partly because you can read the Demis Hassabis story, and the way I tell it in the book really tries to bring this out as sort of a morality tale: a good person who has honest, genuine desire to make the technology beneficial for humanity, who starts by, early on, stipulating that the sale of his company to Google will only go through if there’s a safety and ethics oversight board to make sure the technology is rolled out in a good way, who then fights a three-year legal battle secretly with Mountain View, at Alphabet, to try to enforce that agreement, which Alphabet had kind of pulled back from.

So really, not only talk the talk about safety, but walk the walk. Yet, at the end of my story, he’s locked in this crazy capitalist race between multiple labs in China and multiple labs in the West, where each of them is locked in by the race dynamic. If Demis decided to close down Google DeepMind tomorrow and do no further AI research, it wouldn’t really decelerate things at all because there are so many other labs doing it. So they’re all locked in, and none of them has control of the pace of deployment. That doesn’t feel like a situation in which people in control can really do much.

Inside the Mind of Hassabis (8:29)

In retrospect, it’s crazy. All one can say is that ex ante, the atmosphere in the community of AI builders when Demis began his company in 2010 was that this was a thing that simply didn’t work. AI could not recognize the photograph of a cat. AI could do nothing.

But the reason you were given this kind of access to this company and to Hassabis was just the opposite, so that people would understand the technology and they would, I think, to some degree, understand that at least he was a thoughtful person that you could have confidence would do the right thing so it didn’t take all the jobs and kill us all. But if you became less confident, then that was perhaps a mistake of him giving you this deep access.

Yeah, maybe. I mean, I made a distinction between the fact that he’s good, which I do believe he is, and can he do good, which is a different matter. He admits to me at the end of the book, he’s optimistic still, but without much objective reason for being optimistic because, as I say, he’s lost control of something that he thought at the beginning he could control. Some of the reason he lost control is that the early assumptions were naive. He believed, for example, that there would be one frontier AI lab rolling out AI for all of humanity, with a kind of united team of scientists working together.

A Manhattan project model.

Precisely. Sometimes it’s called the Singleton scenario in the community. Under that scenario, you could be super careful. You could not roll anything out until you tested it 25 times, and maybe you could make it safer. But right now, you can’t do that because everybody’s racing everybody else. Some of the racers are releasing their models on an open-weight basis, where there’s no kill switch for the model once it’s in the hands of a private actor. If the private actor turns out to be a criminal group, you’ve got a big problem. Yet that’s the way we’re headed as a world. So yeah, I mean, you’re right. They hoped to have a story about the reasonableness of Demis, and they got a story about the reasonableness of Demis.

For sure. That was successful. Yes.

Yeah. But there are limits to the consequences of that reasonableness.

I mean, that does sound incredibly naive, that if you think the technology is going to be this powerful, not just the next stage in the digital revolution that began with the PC, and then the PC plus the internet and the smartphone, but something quite beyond that, to not think that the economic incentives would be such that there would be a mad race to develop this technology, keep it progressing as fast as possible, and distribute it as widely as possible; that would be the controlling dynamic, not a 21st-century version of a bunch of brilliant people in a compound somewhere giving out this technology after it’s been triple-checked, in little bits, over a long period of time. I find it remarkable that that was an actual belief that he had.

Yeah, I mean, you’re right. It is remarkable. In retrospect, it’s crazy as well. All one can say is that ex ante, the atmosphere in the community of AI builders when Demis began his company in 2010 was that this was a thing that simply didn’t work. AI could not recognize the photograph of a cat. AI could do nothing. It was deep AI winter. And so, under those conditions, you could assemble the entirety of the world’s strong AI believers in one conference in San Francisco, and it felt like a single community. So, this sort of Singleton scenario, where you just have one lab, was a natural outgrowth of that moment in time. But of course, the moment it started to work, then you were going to have multiple rival attempts to build it because humans are tribal and disputatious and jealous.

The Race for Monopoly (12:31)

I think the agency of the individual was quite strong until 2022, and thereafter the power of the race dynamic takes over.

In the end, then, does it really matter that the company is Google DeepMind and it’s not Meta DeepMind or Musk DeepMind? That the dynamics are such that the dynamics to move forward as rapidly as possible and get businesses to adopt these technologies and use them, that dynamic was going to be the case. Because at one point you mentioned Mark Zuckerberg was maybe going to buy DeepMind, but they like Google more because Google seemed to place a higher emphasis on safety. But in the end, does any of that really matter?

I think it matters a bit. I think there are differences that one can see between the different lab leaders. I would say Mark Zuckerberg is the most purely commercial of the big tech of AI builders. I’d say that Google, because of its strange history, where it had this effectively monopoly product, which just printed money like nobody’s business, a large part of their strategy was to not be too overtly commercial because they were, by dint of where they wound up, almost by accident, already excessively commercial. They were making too much money. So, I have plenty of stories, and I’m sure you know similar stories, about people inside Google five, ten years ago being at meetings saying, “Okay, it’s a bit embarrassing. Our profits are up again. The antitrust people are going to come after us. What are we going to do? Who’s got an idea of how we bury a bunch of money?” Somebody puts their hands up and says, “New research park in Taiwan.” And they say, “You got it.”

Google was sort of post-economic in this sense, and their primary concern, or at least maybe the equal concern, at the senior level, was how do you run this cash-printing machine in a way that is going to be sufficiently socially acceptable that you don’t get killed by government fines? It’s not just EU fines, by the way. Obviously, it’s in the US as well. I think that produces a company which is instinctively respectful of social norms and societal limits. So, when they build data centers today, I know they go out of their way to make sure they’re building new power sources so that there won’t be high electricity prices on the local community, that if the local community wants that new school or that new clinic, they’re going to provide it. I mean, they really do care about building local buy-in for the data center.

So, I think there is a difference between Google and Meta. I’d rather have Google build this kind of technology than Meta. But the bigger picture is everybody’s racing and, ultimately, if you want control, the only person who can impose restraint on all the participants in the race is government.

So, the fact that Hassabis might have an initial different intent or different vision than Sam Altman or Amodei or anybody else is not inconsequential, but certainly, I think, likely to be dominated by just the race.

Yeah. I think there’s a big inflection point in 2022. I mean, if you think about what happened before 2022, Demis had the freedom because he was clearly the leader to define what the next project should be. He chose, at one point, to go and do this protein-folding project. They sunk four years of work into that, and the result was a Nobel Prize-winning breakthrough in structural biology, which is very good for medical advances and maybe material sciences and so forth. That clearly is good for humanity. This is kind of AI with a smiley face painted on it. Whereas once the chatbot went viral at the end of 2022, ChatGPT, then everybody had to pile in and build a competitor, and there’s a lot less leeway to define your own path. So, I think the agency of the individual was quite strong until 2022, and thereafter the power of the race dynamic takes over.

The Economics of AI Anxiety (17:05)

It’s been an extraordinary ride in what is actually less than four years. So, I don’t take back anything I say about the speed of the advance of the frontier. Now that’s different to the speed of the deployment. There I have a lot of sympathy with the economist.

The listeners to this podcast love it when I read. So, I’m going to read another sentence, another beautifully crafted sentence by you here. “The way Hassabis saw things, true general intelligence would make almost anything possible, surpassing the internet, the printing press, even the Industrial Revolution in importance. It would usher in a post-scarcity world of radical abundance resembling the future described in the science fiction he read as a teenager.” And then you quote him: “People aren’t thinking ambitiously enough about what a post-AGI, meaning artificial general intelligence, world will look like.” Now, Sebastian, I work with, and I talk to a lot of economists. That kind of stuff sets their eyes rolling. They just don’t see it. They don’t see it in any kind of reasonable timeline. They think that these technologists, the people in San Francisco, the singularity people, just totally underplay the gap. Even if you’re assuming that these models are super capable in the lab, you are underestimating the gap between that world and the world of these models being used productively.

A lot of skepticism. I’m not the first one to point this out. What are those economists missing? Because you seem pretty optimistic that this is going to be a radically massively transforming technology.

Well, yeah, I think the frontier of technology is accelerating really, really fast. If you look back to 2022 with the arrival of ChatGPT, and then you just think what’s happened since then, you go from a model that hallucinated nonstop, to a model that mostly stopped hallucinating when you plugged in GPT-4 a few months later, to a model with a context window where you could upload the entirety of a Tolstoy novel and query it about the novel. Then you get reasoning, so that logic and mathematics becomes possible. Then you get agentic models. You get these coding assistants. They can handle multimodal inputs and sound and video. The number of improvements, before we even talk about Mythos and the cyber capabilities of that one, I mean, it’s been an extraordinary ride in what is actually less than four years. So, I don’t take back anything I say about the speed of the advance of the frontier.

Now that’s different from the speed of the deployment. There I have a lot of sympathy with the economist. I mean, I think maybe Daron Acemoglu, who gets quoted a lot on this because he’s on the cautious end of his expectations in terms of productivity growth, may be too pessimistic. Having talked to him, I don’t agree with him. I think he doesn’t actually use any of this stuff himself, for example. He doesn’t have a feel for quite how electrifying it can be to use Claude Fable. I think if you don’t use it yourself, your pronouncements should be discounted. So, I don’t agree with him, but I think I would put the mainstream of the economics profession between him and the assumptions you get from the pure technologists in Silicon Valley. In other words, the rollout is going to take a while. It’s going to take a while for corporations to figure out how to make this productive for them. There are certain applications, like writing code, where we are already there. It’s already super helpful. It’s beginning in things like law and call centers and so on. But if you want to reengineer fundamentally how your business is structured and close down departments and all that, we’re not there yet. So that’s going to take a while. It’s instructive to remember how many years ago people said, “Well, there’ll be driverless taxis absolutely everywhere.”

For sure.

It just takes much longer than people expect. But that doesn’t rule out the possibility that superabundance comes in the end. Once you’ve figured out how to use this powerful frontier technology and you’ve deployed it everywhere, maybe you do have a fundamentally bigger GDP number, fundamentally bigger productivity growth rate.

Now I was going to say, “In the end we’re all dead,” but I know that many of these CEOs do not believe that. They believe that death will become optional, so we can get rid of that saying. But not to belabor the point, but this is not radically new economics, the notion that it takes a while for technology to be effectively implemented into the economy. Yet, you have CEOs, until very recently, where I think the message has gotten through, saying that everybody who works behind a computer will be out of a job in 18 months. Now, whether that’s exactly what he meant, that is certainly how it was interpreted. So, is it any wonder that people are going to oppose data centers? Why should you even take the environmental risk, or the risk of higher electricity prices, for a technology that will basically destroy the world as I know it, my life, in 18 months?

I think this has been incredibly reckless by these CEOs. Demis Hassabis has been one of the more cautious people about timelines, though I will say that even he has sort of, I think, brought his timeline up. Though he does not make those kinds of predictions, I don’t think.

He doesn’t say that 50 percent of entry-level jobs will be disrupted in five years, or by 2030, which is what I think Dario Amodei did say. So that is a distinction between them. I agree that I think Dario’s projection on that particular point is not credible. I think he’s way exaggerating how quickly that changes. Even though I am inclined to believe him when he says that there’ll be recursive self-improvement in terms of frontier model development inside Anthropic by the end of 2028, he knows that much better than I do, and I have no reason to dispute his forecast in terms of the advances in computer science. It’s just the rollout that I don’t agree with him on. I think there are lots of examples. Law is one thing which is projected to be disrupted enormously because you’ve got these cognitive workers sitting at a screen, and that can be automated.

But when I look at the corporate law firm at which my son is employed in London, they’ve always hired 80 bright young kids every year, and they’ve already always gotten rid of 65 out of the 80 by six years later. They always overhire. Why? Because it’s nice to be able to pick the best ones, but also it’s good for network entrenchment. When you kick them out, they go work at your clients, and then when they want business later, they turn around and hire the law firm where they got trained. So there are all kinds of reasons why people will hire young graduates. It’s not as simple as saying, “Oh, because there’s an AI you won’t hire anymore.”

Governing the AI Race (24:17)

People often think of AI as a bunch of code that flies around cyberspace, and you really can’t control it. But actually, it’s also a bunch of data centers which are huge physical installations. The government knows precisely where they are. They can’t be moved or hidden.

From what you’ve gathered, what sounds more difficult? Building a super intelligence that will not do things we don’t want it to do, or is it more difficult to regulate such a superintelligence if this is indeed a rapidly evolving, powerful technology, easy-to-copy, globally distributed, huge economic incentives to keep it moving forward? I guess, to phrase it another way, are efforts at regulation basically going to be doomed? We’re going to talk about your P(doom). This is the R(doom), the regulation doom.

Yeah. I’m actually quite optimistic in terms of the ability of a government agency to regulate. I know that’s perhaps not a fashionable thing to be telling an AEI podcast. But here’s my thinking. People often think of AI as a bunch of code that flies around cyberspace, and you really can’t control it. But actually, it’s also a bunch of data centers which are huge physical installations. The government knows precisely where they are. They can’t be moved or hidden. The owners are known, the payment flows can be discovered. And you can go, as the government, you can go tell those data centers, “Listen, you’re not going to train the following model. And if you do, I’m shutting you down.” There is a choke point in this technology. So please, first of all, get the kind of code flying around cyberspace, quicksilver, impossible to track down. Just please, let’s park that myth because actually there is a choke point.

And then, once you understand that there’s a choke point, the next thing is: don’t let models out into the wild without there being a kill switch.

That’s another way of saying it doesn’t allow open-weight models. To me, it’s a no-brainer that we’re going to stop open-weight models. I mean, the Trump administration has just done a 180 since Mythos appeared on the scene and is controlling the release even of proprietary models. So the notion that they would control, or even forbid, open-weight models, which can be freely adapted by the user and can never be shut down, of course they’re going to do that. Would China ever do it? Well, there is a Reuters story this week saying that they were thinking about it, and I don’t know where they are. I don’t know if the Reuters story is correct, but the logic in terms of the direction of travel is surely that China loves controlling stuff.

They’re absolutely happy to regulate the internet, to tell their teenagers they can only do X amount of video games per day, whatever. They have no problem intervening. They’re going to say to themselves, “Do we want Mythos-level cyber-hacking technology to be freely circulating in our country so people can do cyber-attacks? Of course not.” They’re going to shut it down too. Now the question is, will they shut it down in terms of their models being used outside of China? They might be quite happy to destabilize the West with open-weight models, which is why I think we need to talk to China and try to do a nonproliferation deal with them. But I do think that the government, fundamentally, can regulate data centers, forbid open-weight releases. And it’s some combination of a sort of Food and Drug Administration model, where before the drug is released, it has to be tested and you can veto the release if it’s not a safe drug.

Then Fed supervision, where, on an ongoing basis, you might have government officials embedded at the major AI labs, the major AI-using companies which have some systemic risk factor because they are running big operating systems on the internet, or whatever it might be. You’d have ongoing supervision by government agents to try to reduce screw-ups. Now, I’m fully aware that Fed supervision often fails, but I think it might be better than nothing. But what I’m saying is that I think there’s too much defeatism around.

I feel like all that’s unlikely to happen within the next 18 months. If you listen to some aggressive forecasts, we have 18 months to do everything you just said.

Well, you might be right, but let me put it to you this way. In March of 2026, if you had asked anybody in the Trump administration, “Are you going to ban the release of a model and then, case by case, restrict which consumer is allowed to buy it?” they would’ve said, “No way, we are absolutely in favor of accelerating AI rollout.” By late April, they’d flipped. They did a 180. They did something that they didn’t predict they would do. So I think the capacity for the technology itself to create facts on the ground which oblige government to respond should not be underestimated.

I think about nuclear nonproliferation, which may be regarded as a success, but yet North Korea still has nuclear bombs, and I think Iran could get a nuclear bomb quite easily. I’m wondering if that was one way you could view that as not successful. There are certainly a lot of choke points with building a nuclear weapon. I wonder whether, at the end of the day, North Korea doesn’t have its own superintelligent model just like everybody else does.

Yeah, but on the other hand, when was the moment when Abdul Qadeer Khan of Pakistan stole the blueprints from Holland or Denmark, or whatever it was? 1990 or something, right? So there was an operative nonproliferation regime roughly from 1970 to 1990. 20 years is better than nothing. I’m not in favor of giving up before we try.

The Biggest Leap Since Abstract Thought (30:13)

I just ask you, spend a couple of hours with Claude Fable and then see if you disagree with me… I think it is smarter than me by quite a long shot on any topic I ask it about.

The quote I read at the start, “At some point in the not-distant future, AI will beat human intelligence and be the biggest thing since we acquired the capacity for abstract thought”—people question that. When do you think that will be undeniably true, that sentence about what this technology is and what it can do will be obviously true beyond dispute?

To be honest, I would say it’s already true. I mean, you try using Fable, and if people are listening and they’re inclined not to agree with me, I just ask you, spend a couple of hours with Claude Fable and then see if you disagree with me. Claude Fable is an astonishing jump from the capability that existed a month or two ago. And to me, when I ask it something, I literally put this in. I said, “Design me a tax reform which seeks to balance three different objectives in the United States, which are a desire to slow displacement of workers by AI, but not decrease frontier innovation in AI, and protect the government’s ability to collect taxes in an environment where individual labor may be a smaller part of GDP.” It produced a very sophisticated policy proposal, way more sophisticated than I could have come up with if I’d had a week, and it did it in 10 minutes.

So, I think it is smarter than me by quite a long shot on any topic I ask it about.

Let me flip it around and ask you this way. Looking back seven or ten years from now, what are the odds that we’ll view your book rather than one documenting the rise of superintelligence, but rather one documenting a mindset and hubris that led to the biggest investment bubble, and maybe letdown, of all time, that does not turn out to be the last invention ever invented? Is there a nonzero percent chance that that will be true? That is the book you have, in fact, written, documenting hubris rather than success.

Well, look, you know as well as I do that there are lots of cases where you have a railway bubble. It doesn’t mean that the railways are not a very important technology. So, I’m not saying there could never be a bubble in AI stocks. I think that OpenAI, as an individual company example, is way overvalued. For example, I don’t think it’s got a business model. It’s got great technology, but no business model that’s going to sustain the valuation that it’s engineered for itself. So, I’m willing to say that, of course, markets can overshoot. I don’t think that negates the forecast about the technology being really deeply transformative.

So, you believe there is basically a zero percent chance that this is not a transformative technology, whatever the impact on markets and stock prices. Setting all that aside, this is a transformative technology, and we can begin to plan based on that assumption.

Yeah. I mean, I think a very lowball assessment, the bottom end of my range for plausible judgments about how important this is, would be to say it’s like the internet plus the first ten years of applications. Which is big. That’s the lowest I can imagine being the case.

What does a good outcome here look like? I’m very good, and Hollywood is very good, and people who don’t like data centers are very good at pointing to how this doesn’t turn out to be something that will have been worth it and makes our lives better, and makes not just people live in rich countries, but everybody’s lives better. What does the good outcome look like?

Look, I think the most obvious upside is in science, medical discoveries, and material science discoveries. The latter may be leading to breakthroughs in energy generation that fundamentally reduce the speed of climate change. So, I think that’s the biggest upside.

Discussion about this post

Ready for more?