Two Failure Modes of Emerging Technologies

Some thoughts on AI and sociotechnical thought experiments.

There is a pervasive sense right now that, in the field of artificial intelligence, we are living in early times. Depending who you ask, AI is some mix of exciting, inevitable, and scary. But all agree that it is real, it is here, and it is growing. The present is merely prelude. The future is about to arrive.

When imagining the trajectory of an emerging technology, there are two types of thought exercise one can engage in. To put it plainly, one can imagine what might happen when a technology works as advertised, but at larger scale, or one can imagine what would happen if a technology breaks down, or does not work as well as advertised.

I refer to these the two failure modes of emerging technologies. Both exercises have value, but we should be wary about conflating them.

In recent weeks, there has been an outburst of warnings that fall within the first failure mode. In a recent opinion piece for the New York Times, Yuval Harari, Tristan Harris and Aza Raskin likened AI to an “alien intelligence” that we have recently collectively summoned. The tone of the piece is overwrought, pleading, hair-on-fire stuff. The authors warn:

A.I. could rapidly eat the whole of human culture — everything we have produced over thousands of years — digest it and begin to gush out a flood of new cultural artifacts. Not just school essays but also political speeches, ideological manifestos, holy books for new cults. By 2028, the U.S. presidential race might no longer be run by humans.

Then there is the Future of Life Institute open letter, calling for a six-month pause in the development of generative AI models. The letter warns that the extraordinary pace of AI development is risky — that we are collectively not ready for the profound societal changes that are about to be unleashed. It was signed by dozens of major figures and serious thinkers in the AI research community, and also by Elon Musk.

While I agree with the remedy — I too would like to see a pause in the pace of AI development — I also worry that Harari, his coauthors, and the open letter signatories are focused on the wrong failure mode. And that has consequences for the sorts of remedies that are discussed and the options that are foreclosed.

The core problem isn’t that we are on the cusp of artificial general intelligence (we aren’t); the core problem is that we have an over-inflated sense of what these generative AI models are useful for.

(Brian Merchant offered a similar sentiment in his recent column, “I’m not saying don’t be nervous about the onslaught of AI services — but I am saying be nervous for the right reasons.”)

Ben Collins @oneunderscore__

I'll say it: The AI panic is starting to feel like an advertisement.

---

One way that a technology can fail is that it can work as intended, but at a much larger scale, with unexpected results. As a guiding example of this failure mode, consider Apple’s AirTags. The basic use-case for the technology is simple enough: (1) attach a sensor to your keys, (2) When you lose your keys, you can find them with your phone.

(Other companies established the market for this use-case. Then Apple unveiled a competing product that was built into the broader IOS ecosystem.)

But, as the market grew and AirTags became affordable and easy-to-use, an a second use-case emerges. It turns out this is a handy tool for creepy dudes who would like to (1) toss an AirTag into an unsuspecting woman’s bag, then (2) behave in even creepier, more dangerous ways.

This type of knock-on (mis)use case could have been forecast in advance. (The idea of placing trackers everywhere sounds great to comfortable white guys, but raises immediate red flags for basically everyone else.)

To the company’s credit, last year Apple recognized this problem and responded by adding more friction into the product. So long as you are part of Apple’s (expensive) product ecosystem, you now receive an alert if an AirTag is following you that is not registered to you. That is far from a perfect solution. A lot of harm can still be done to a lot of vulnerable people. But it’s more than nothing.

Thinking in terms of this first failure mode can be important for product designers and policymakers. It invites us to contemplate “let’s say this technology works as-intended. What new problems and opportunities is it likely to create? What choices might we make to mitigate the harms and amplify/better distribute the benefits?”

The first failure mode also strongly resembles the type of thought experiment that can make for very good science fiction. But it can often descend into criti-hype. It tends to treat technologies as natural forces, as though AI (for example) were a hurricane.

Assuming that an emerging technology works as-intended can be a huge stretch. These are frequently developed through a mix of gumption, personal charisma, and a compelling powerpoint slide deck. The culture and values of Silicon Valley and its venture capitalist masters does not reward “measure twice, cut once,” showing-your-work, or scientific peer review. It is a culture of “move fast and break things,” a culture of lying in your marketing materials and paying the fines later if you happen to get caught.

Believing and amplifying those marketing claims is often a sucker’s bet.

---

The second failure mode prompts an entirely different set of questions. What if the bugs in the emerging technology are not resolved? What if it the market for it grows, and it gets incorporated into critical social systems, but it continues to fail in ways that are increasingly hard to see?

The most obvious example is the rise and fall of Theranos. The problem with Theranos was that the technology was deployed to a mass consumer base for critical medical testing, but it simply didn’t work. Elizabeth Holmes and Sunny Balwani made a bet that with enough time and money, they could eventually deliver on their technological promises. It’s a bet that plenty of successful Silicon Valley companies have made. And the bet paid off for years, until the regulatory system for medical devices shut it all down. (Tech companies brazen past regulators all the time. But some regulators are easier to brush off/intimidate/buy off than others. If Holmes and Balwani had worked in fintech or adtech, they wouldn’t be in prison today.)

The Theranos comparison is too easily dismissed, though. Their blood-testing machines didn’t work. I typed “two computers, both on fire, cyberpunk digital art” into OpenAI’s free Dall-E2 engine. It produced the picture at the top of this post within seconds. That is, quite obviously, far more real than Theranos’s product ever was.

Better perhaps to recall the promises that were made just a decade ago, when it appeared we were in the early days of the “age of big data.” We are still living with the results of that mess, and they fall squarely within the second failure mode.

Consider, for instance, the examples provided in Virginia Eubanks’s 2018 book, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. Eubanks examines how automated systems were used by state-level government agencies in an attempt to improve service delivery (and lower costs). The results will leave you wanting to smash your computer.

The problem with automating government service delivery is not that it works too well, leading to unintended consequences. The problem is that the data is crap, the inferences drawn from that data are crap, and all that crap is obscured from public view. The problem with “big data,” in other words, turned out to be that the products did not work nearly as well as advertised. Misuse causes harm. And, in the rush to modernize social services, vulnerable people are left much worse off.

(And lest you think this is old news, make sure to read last month’s WIRED investigation of algorithmic welfare mismanagement in Rotterdam. We are living with the aftereffects of the “Big data” hype bubble.)

It’s easy to overlook this second failure mode, because we so often assume that today’s technical problems are only temporary. Particularly when a technology seems to be improving by leaps and bounds, it can appear shortsighted to focus on all the benchmarks it hasn’t yet surpassed. Tech companies iterate and refine. A cultural faith in Moore’s Law anchors the belief that today’s technical limitations are only temporary.

That’s a mistake. The closer you peer into any sociotechnical system, the more complicated and prone to failure it turns out to be. And these are socio-technical systems. We adopt new technologies not because they objectively work better than their predecessors, but because of a range of social incentives that shape the behavior of powerful actors and institutions. Often the companies that build these technologies hide behind intellectual property laws and trade secret protections to prevent any clear evaluation of whether their product works as-intended. Whole industries can be propped up on the motivated reasoning and shared delusion that the product must work as intended, simply because of the scale of its use.

We ought to pay more attention to the second failure mode when imagining the trajectory of AI. I’m not worried about an imminent future of artificial general intelligence. I’m worried about a future where generative AI tools get baked into social systems and wreak havoc because the tech doesn’t work nearly as well as intended.

---

Years ago, I had a student who wanted to write a semester paper on Palantir. The student, a fierce leftist, was convinced that Palantir was evil. The company had become a critical component in the digital surveillance state. It aided the war machine, at home and abroad. It helped drones identify targets in the Middle East. It helped cops harass black people in the U.S. And, also, its surveillance and intelligence models sucked. The company routinely overpromised and underdelivered. It couldn’t do nearly as much as it claimed in its marketing materials.

Throughout the semester, I pushed him to clarify this tension for me: was Palantir a malevolent force because its products worked so well, providing aid to a set of social institutions that oppress and harm vulnerable segments of the public? Or was Palantir a malevolent force because its products didn’t work as intended, while creating a near-impenetrable veil of mathematical certainty that shields those institutions from scrutiny?

These are fundamentally different criticisms, built out of the two distinct failure modes. Conflating them can be a bit like the proverbial jewish grandparent kvetching about a restaurant (“the food was inedible. And the portions! So small!”).

Consider: we live in a world where the United States conducts some percent of its foreign policy through drone strikes. One moral case one could make against Palantir is “this is immoral, and your technologies are part of it!” But the company would surely reply with a conditional defense — if the U.S. is going to conduct drone strikes, then isn’t it for the best that these strikes be precision-targeted? This is the type of ethical debate that failure mode 1 invites us to entertain.

By contrast, it is unquestionably clear that if you are going to deploy predictive models to help identify likely terrorists to be drone targets, then you damn well ought to be certain that your predictive model is correct! …And sure, no system can ever guarantee 100% accuracy. But the minimum moral threshold still must be “fewer errors are made by this new, automated system than were made by the system it is replacing.”

How would we evaluate if this were the case though? We would, at a minimum, need data transparency and independent algorithmic audits. We would need clear industry standards, harsh penalties for violating those standards, and accolades for those who exceed them. Otherwise, companies would surely hide their mistakes, shield themselves from oversight, and try to compromise all transparency and auditing processes.

Would you care to guess what sort of data transparency and algorithmic auditing standards actually exist in the United States? (I’ll give you a hint: it’s a fucking nightmare.)

It is certainly worth imagining what sort of world will be created if companies like Palantir build products that work as-advertised. But I think we should be at least as concerned about the potential harms that arise when their products don’t work nearly as well advertised.

---

Amazon.com has some of the best data of any company in the world. It is a massive, profitable corporation, with years of granular data on what we search for, what we read, and what we purchase.

Based on my user profile, Amazon sends me a lot of emails, encouraging me to buy more books. Probably once a month it sends an email like this one:

That’s The MoveOn Effect: The Unexpected Transformation of American Political Advocacy. I have never purchased that book through Amazon. I am extremely unlikely to buy it through Amazon, though. And that’s because I’m the author of the book.

I’ve always found that amusing. Amazon — a company famed for converting user data into profit, has never connected the dots between David Karpf, the reader and David Karpf, the author.

I mention this because two weeks ago I took Google Bard out for a spin, and I decided to ask it to summarize The MoveOn Effect. Bard immediately fabricated a book with the same title/different subtitle, attributing it to an author named David Callahan, and made up a bullet-point summary that has little to do with the actual book as-written.

I checked it out. Callahan never wrote such a book. Callahan, as far as I can tell, has never written about MoveOn.org or digital activism. Google Bard “hallucinated” the book entirely.

The simplest explanation is that my book is a little too obscure for Google’s Large Language Model (Ouch. …But also, that’s fair.). GPT4, with its several billion parameters, might have enough data to identify the existence of my work.

Still. I’ve been hearing for months that ChatGPT and similar LLMs are the end of higher education. And yet this is already the second time I have personally run into this phantom citation problem. These tools aren’t nearly as well-suited to the task as their loudest proponents and detractors claim.

---

Let’s grant that we are living in early times with AI. How, then, should we proceed?

The Future of Life Institute open letter treats AI through the lens of the first failure mode. It warns that we are on the cusp of artificial general intelligence (AGI), and asks that we slow down the pace of development so society can prepare. It makes for interesting science fiction. But it also promotes the very companies and products that it sets out to criticize.

Missing from the letter is any consideration of the second failure mode. The biggest threat from generative AI right now is that, in the rush to construct new business models and win the race to dominate the technological future, we will build these technologies into systems for which they are terribly suited.

Maybe you’ll have an AI doctor someday. But will that doctor be better or worse at diagnosing medical conditions than your existing doctor? Maybe AI will write first drafts of your essays. But will it “hallucinate” key claims? Do we have any of the transparency and auditing systems in place that would be necessary for us to even know?

We ought to slow down the pace of AI development — not to prepare for the inevitable triumph of AGI, but to get a better sense of the technology’s limitations. The best way to reduce potential harms is to stop treating the technology like magic.

I’ve argued elsewhere that, over time, the trajectory of any new technology bends toward profit. Slowing down the pace of AI development is a great idea. The best way to do so is by adding reasonable friction into the potential revenue streams.

I suspect Yuval Harari and his coauthors, along with most of the signatories to the Future of Life Institute letter, are not eager to add friction to the potential revenue models for these products. That isn’t the type of policy response that fits into the thought experiment they are engaged in.

But they ought to. We all ought to. If we don’t take seriously the ways that a novel technology can fail by not living up to its promises, then we ignore a whole class of entirely-predictable social problems.

One of the chief lessons that has come out of my study of the history of the digital future: the technologies of the future suffer from all the same shortcomings as the technologies we have today.