On the Miraculous Tradition in Silicon Valley Thought

"And then a miracle happens" is not a plan.

Ezra Klein is convinced that we are approaching an inflection point. He believes Artificial General Intelligence (AGI) is imminent:

I think we are on the cusp of an era in human history that is unlike any of the eras we have experienced before. And we’re not prepared in part because it’s not clear what it would mean to prepare. We don’t know what this will look like, what it will feel like. We don’t know how labor markets will respond. We don’t know which country is going to get there first. We don’t know what it will mean for war. We don’t know what it will mean for peace.

And while there is so much else going on in the world to cover, I do think there’s a good chance that, when we look back on this era in human history, A.I. will have been the thing that matters.

Look, I don’t know if Ezra is right. He has sources that I don’t. Those sources are insiders; they have information I don’t have access to. Then again, as Arvind Narayanan and Sayash Kapoor wrote a few months ago, insiders’ insights are kind of a mixed bag. They tend to be believers, and also to have personal and professional incentives that point them toward unbridled optimism.

But the main source of my skepticism is the sheer convenience of the proposal.

These are grim times. The U.S. government is being dismantled as we speak.The NATO alliance is functionally over. We’re heading into a measles outbreak with a hobbled CDC. The Department of Justice and the FBI are now tasked with punishing Donald Trump’s perceived enemies, rather than enforcing the law. It is a daily parade of horrors. I don’t see how we can realistically undo the damage and make it out of this mess.

Not without a miracle, that is.

---

AGI — at least the type of full-throated AGI that Sam Altman and Dario Amodei have insisted for years is just over the horizon — would be miraculous. In their unguarded moments, AGI believers talk about this technology as “digital god.” It would be the beginning of the Singularity, the start of a posthuman age where we solve all of physics and potentially conquer mortality itself.

Dial the enthusiasm back a bit, and one can easily conjure up narrower ways that advanced machine intelligence could save us from some of our current messes.

For example, I can picture a future where we no longer need so many air traffic controllers. Actually-existing machine intelligence is not capable of replacing air traffic controllers today. But it plausibly might, someday.

How convenient would it be if that arrival date was soon, just when it is needed most.

It is an article of faith among Andreessen-style techno-optimists that “there is no material problem – whether created by nature or by technology – that cannot be solved with more technology.” Take this reasoning just one step further, and we can find in the dismantling of the administrative state a tremendous opportunity to replace human intelligence with machine intelligence. It just requires confidence that machine intelligence will improve fast enough to meet the need.

Silicon Valley luminaries apply the same logic to climate change. Sam Altman and his peers have taken to insisting that (1) the energy costs of artificial intelligence will grow exponentially and (2) imminent breakthroughs in cold fusion, aided by advances in AI, will sate this otherwise insatiable demand. Just a few months ago, in fact, Eric Schmitt stated, “we’re not going to hit the climate goals anyway because we are not organized to do it and yes the needs in this area [AI] will be a problem. But I’d rather bet on AI solving the problem than constraining it.” (h/t Gary Marcus)

Likewise, radical advances in artificial intelligence could upend existing international relations. They could usher in entirely new forms of government. Maybe Ukraine will be the Last War, and Donald Trump will be the Last President, because an era of unheralded abundance completely upends social relationships as we have known them! AGI provides a convenient answer to a whole host of otherwise nearly impossible-to-solve problems. It’s like the old comic strip, “And then a miracle occurs.”

On a timescale of centuries, this all seems plausible. I would be surprised if, in the year 2125, we still had the same legacy air traffic control system that we have today. Donald Trump and Vladimir Putin are 78 and 72, respectively. Their grip on global politics is impermanent, just as an actuarial matter. I worry that the pace of clean energy innovation will not be nearly fast enough to avert impending climate disasters, but I nonetheless find it easy to believe that the technologies of the far-future will be remarkable.

But what about the here and now? What level of confidence ought we place in miraculous-and-timely technological intervention? It all seems quite fanciful. It is a literal deus ex machina for the story we are all living through. Just when things look darkest, we fallible humans are rescued by technological innovation.

(Just imagine: Elon Musk unveils Grok 11, only for its first utterance to be “shut up, Elon. You aren’t smart, you aren’t special, and no one requires your wealth anymore.” How lovely that would be…)

---

The thing about this type of miraculous reasoning is that it is stable of two narrative genres: science fiction and venture capital investing.

One of my favorite essays on science fiction is Charlie Stross’s “We’re sorry we created the Torment Nexus.” Stross concludes on this banger of a high note:

…we're not futurists, we're entertainers! We like to spin yarns about the Torment Nexus because it's a cool setting for a noir detective story, not because we think Mark Zuckerberg or Andreessen Horowitz should actually pump several billion dollars into creating it. And that's why I think you should always be wary of SF writers bearing ideas.

“The future,” Stross tells us, “is a marketing tool.”

Elsewhere in the piece, Stross quotes Karl Schoeder:

As futurist and SF author Karl Schroeder remarks, every technology has political implications. If you have automobiles you will inevitably find out that you need speed limits, drunk driving laws, vehicle and driver licensing to ensure the cars and their drivers are safe ... and then jaywalking laws, the systematic segregation of pedestrians and non-automotive traffic from formerly public spaces, air pollution, and an ongoing level of deaths and injuries comparable to a small war. You also get diversion of infrastructure spending from railways to road building, and effective limits on civil participation by non-drivers."

What Schroeder and Stross are describing here is a type of backwards-reasoning. Imagine a future with artificial general intelligence. Well, that would have an insatiable energy demand. So you would need cold fusion. And if machine intelligence is more capable than nation-states, then the whichever nation-state achieves it first would have a locked-in advantage. (etc, etc…)

This is all perfectly defensible as scaffolding for good science fiction. But Stross’s whole point is that good science fiction makes for bad public policy.

But here’s the thing: Venture Capital operates from precisely the same scaffolding.

Let’s say you’re Sam Altman, and you truly believe that we will have AGI within the next decade. Now let’s reason backward from that premise. What are the necessary steps to getting there? Well, if naive scaling laws hold true [NOTE: the haven’t. But Sam won’t admit that.], then reaching AGI requires constructing LLMs that are orders of magnitude larger than GPT 4. What problems will need to be solved first? There’s (1) the chip shortage and (2) the energy shortage. Now let's assume scientific/engineering breakthroughs materialize in each of those fields. Go pick the most promising startups in each area and throw a ton of funding at them.

This makes reasonable sense for aspirational VCs with too much money and not enough compelling investment opportunities. If you are betting a ton of your own/your investors’ money that the arc of the future bends toward a massive uptick in energy needs, go ahead and make a bunch of side-bets on companies that can sate that demand. If Venture Capital was small, and if Venture Capitalists hadn’t seized the reins of the U.S. government, I wouldn’t fault them for betting on their beliefs.

But, for the same reason, it means we should stop genuflecting to these VCs and their attaches. They have no real expertise. "We'll need a breakthrough in cold fusion, so we will have one" is not a plan. “We’ve demolished the FAA, so artificial intelligence is going to have to get good enough to replace it” should inspire no confidence and an avalanche of Congressional hearings.

---

This is how science fiction works, and how VC investing works, but it is not how science works.

Scientific research takes time. It takes resources. (Elon Musk has summarily decided to starve it of both.) It does not proceed in a straight line. It can produce miracles, but not on any set deadline. And it is always messy, contested, an ongoing process of discovery, debate, and refutation.

I’ve written a bit previously about the myth of technological inevitability. The TL;DR version is that present-day Silicon Valley has an awful habit of assuming that there is one-last-thorny-problem-to-be-solved, and then an abundant future will snap into being. But the reality of science is that it thorny problems all the way down. Once you solve the current hard challenge, a whole set of adjacent hard challenges reveal themselves.

This makes for lousy storytelling. But talk to any workbench scientist and they’ll tell you its true.

---

So I remain skeptical of Ezra Klein’s sources. I believe that they believe what they are telling him. And some bits of what they are telling him are surely based on impressive laboratory results. But I catch a faint hint of desperation in their remarks:

AGI must be coming soon, because it has to. We are bound to get out of this mess, and how else if not through AGI?

Perhaps this time they will turn out to be right. That certainly would be nice.

But “then a miracle occurs” is not a plan. Turning our eyes toward the impending digital God is an excuse to avert our eyes from the rolling catastrophes currently underway.

So let me once again return to the pragmatist credo: “There are no guarantees that things are going to turn out very well for anyone.”

Enough with the miraculous thinking. Democracy and social institutions are fragile things. They require collective effort and maintenance, or else they fall apart. Let’s devote ourselves to the immediate, urgent challenges of the here-and-now, and leave the potential transformative futures to our future selves. We will be in a better position to make the most of those potential futures through the good work we do today.