🎆 A serious paper about the path to superintelligence

The folks at Google DeepMind sketch out a path forward

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

It was Nobel Prize–winning physicist Niels Bohr, or maybe science-fiction author Arthur C. Clarke, or even possibly New York Yankees catcher Yogi Berra—or perhaps all of them, in one way or another—who neatly observed that predictions are difficult, especially about the future. That paradoxical witticism, for lack of a more elegant term, lies at the heart of the new Google DeepMind paper “From AGI to ASI.”

Like those other guys (maybe?), the fourteen authors—including Shane Legg, the company’s cofounder and chief AGI scientist—concede that “the future is unpredictable.”

But there's more to the story, which is why the paper is so valuable. Because the pace and nature of AI progress are unpredictable and because the stakes are massive, the researchers think it's a good idea right now to map out the possibilities of "how AI itself might continue to develop in a post-AGI world along the continuum of machine intelligence."

Lots of uncertainty, although the researchers claim they are pretty sure about one thing. As they explain:

Perhaps, reaching human-level AGI will take longer than a few years. What can be said with certainty is that even if AI progress continues far beyond human-level AGI, this does not mean that ASI will be omnipotent, and that ASI will certainly be able to “cure” ageing, reshape matter arbitrarily with nanobots, upload human brains, build Dyson spheres, or restore the planet’s climate and bio-diversity to pre-industrial levels. Either way, predictions when AI progress plateaus, and at what capability level, will remain difficult and uncertain.

I dunno. That passage being included actually kind of makes me think these folks may well consider artificial superintelligence, or ASI,¹ as being quite capable of all that science fictional stuff. After all, Google DeepMind CEO Demis Hassabis thinks advanced AI will be able to perform some absolute techno-marvels, as documented in The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence by Sebastian Mallaby.

From the book:

The way Hassabis saw things, true general intelligence would make almost anything possible, surpassing the internet, the printing press, or even the Industrial Revolution in importance. It would usher in a post-scarcity world of radical abundance, resembling the future described in the science fiction he had read as a teenager. “People aren’t thinking ambitiously enough about what a post-AGI world will look like,” Hassabis once told me. “I still hear people talking about the limits to our resources. Like, will we have enough to pay for government programs to deal with the fallout from AI, such as a universal basic income? Or for the electricity to power the data centers? But it’s going to be like Iain Banks’s Culture series. We’re going to be mining asteroids. We’re going to solve nuclear fusion. We will have ways of extracting hydrogen fuel from seawater. People are not understanding the magnitude of the change.

---

Talking timelines

OK, so that's the dream. But when?

Earlier this month Hassabis said AGI was a sooner-rather-than-later thing, “maybe 2030, plus or minus a year, which is astounding to think, really. I think that will be such an enormous transformative technology; it’s gonna effectively be a new human era.”²

And that’s assuming, by the way, we can science our way through various remaining problems, ones Hassabis typically clusters together as the need for “one or two more big breakthroughs,” which he expanded upon thusly during the Big Technology podcast back in January:

“I’m definitely a subscriber to the idea that maybe we need one or two more big breakthroughs before we’ll get to AGI. And I think they’re along the lines of things like continual learning, better memory, longer context windows—or perhaps more efficient context windows would be the right way to say it—so, don’t store everything, just store the important things. That would be a lot more efficient. That’s what the brain does. And better long-term reasoning and planning.”

But remember, we're talking about getting to ASI, which is even harder but hardly impossible. As touted in its title, the paper highlights four possible routes—which could work in unison—from AGI to something far beyond it.

First, “effective compute” may keep rising extremely fast—perhaps roughly tenfold a year—thanks to better chips and bigger chip budgets, plus software that squeezes more out of each one. Second, there’s the wisdom of AI crowds. Even if individual models stall around human-level intelligence, running vast numbers of them—superfast, 24/7/365—and coordinating them could create something far more capable. (The Borg knew what they were doing.) Third, a flywheel of progress. AI could begin helping build better AI, creating a self-reinforcing loop that could tip into an intelligence explosion. Finally, some significant breakthrough could change the game. It could be a new architecture or training method, not just a bigger version of today’s systems.

The good news: “Taking all of this together, we believe that the possibility of cruising past AGI and into ASI territory within the next decade or two cannot easily be dismissed.”

---

Looking at limits

None of this is guaranteed, though. Doing something that has never been done before will no doubt have problems both foreseeable and impossible to foresee. Scaling could hit the wall of economics and physics—only so many chips, so much electricity, so much money. Did I mention electricity? AI-making-smarter-AI still has to work properly in the real world, and maybe the self-improving feedback loop fades as easy gains run out. There’s no rule flywheels have to keep spinning. Perhaps Team Agents gets bogged down in a bureaucratic quagmire. Also: Counting on big breakthroughs isn’t much of a strategy, really.

And some of the most plausible friction isn’t technological at all: Governments—nudged by a spooked, Down Wing public—could simply decide to slow the whole thing down.

Further, getting to ASI may require something extra, a truly miraculous bit of technological sparkle: the ability to look at today's reality and pull out a genuinely new idea. Such cognitive leaps are what power scientific revolutions. Rearranging known concepts, no matter how cleverly done, is insufficient.

Think of the paper's thought experiment. Take an AI and train it on everything humanity knew about physics before Newton, then ask it to produce general relativity like a techno-Einstein. The authors are skeptical: "It seems highly improbable that the system could reason its way to the laws of general relativity, let alone quantum mechanics, while lacking the conceptual primitives of calculus, universal gravitation, or electromagnetism."

Today's models inherit their concepts from us. Whether a machine can extract brilliant new ones from the world remains, the authors suspect, unsolved.

Which leads us to a robust research agenda, from mapping the bottlenecks more precisely (think Google Street View) to pinning down how self-improvement loops actually behave and what superintelligence would theoretically require. From the conclusion: “And while we can only see a short distance ahead, we can see plenty there that needs to be done.”

Faster, please!

1. “AGI denotes a system that reaches at least median human performance on a very broad set of cognitive tasks. ASI, in contrast, refers to a system that has general superhuman intelligence, meaning a system that outperforms large groups of (thousands of) human experts that work over an extended period of time (years).”

2. It’s my recollection that the typical Hassabis speculation has been that AGI could emerge over the next five to ten years. Indeed, he used that exact framing, for instance, in March 2025 when speaking to CNBC (where I am an official contributor): “AI that can match humans at any task will be here in five to 10 years, Google DeepMind CEO says.”

---

On sale everywhere by James Pethokoukis The Conservative Futurist: How To Create the Sci-Fi World We Were Promised