Are large language models on the trajectory of word processing or digital advertising?

(In which Dave stumbles upon another analogy.)

In his newsletter this week, Steven Levy looks at the trajectory of ChatGPT and sees echoes of VisiCalc. VisiCalc was the “killer app” of personal computing, the forerunner to a transformative general purpose technology. And Levy has been a tech reporter for four decades. He’s one of the best in the business. He even covered VisiCalc back in ‘84.

Levy is reacting to the latest Pew Internet & American Life survey, which shows a steep and steady increase in people using ChatGPT in the workplace (8% last March, 12% in July ‘23, 20% in February ‘24). I don’t have a ton to say about the poll. Like, it’s fine, but it’s also just a single survey question. It calls to mind the Pew data on internet adoption from the late ‘90s and early ‘00s. That data was comprehensive-but-thin, and I spent a chunk of my early career criticizing how it was over-interpreted everywhere. It’s a useful data point, and all we have to go on right now, but I’d caution against putting too much weight on it.

I find the VisiCalc comparison evocative, though, because I’ve been fiddling for a couple weeks with a similar analogy: Microsoft Word. It seems to me that a lot of AI converts are convinced that the ChatGPT is on par with word processing in the early ‘90s.

(Spoiler alert: I’m gonna disagree.)

An Apple IIe running Visicalc. Image from ComputerHistory.org

---

I’m drawing here from a delightful James Gleick essay from 1992, “Chasing Bugs in the Electric Village.” It was originally published in the New York Times, and later republished in his 2002 book, What Just Happened?: A Chronicle from the Information Frontier. Gleick describes the excitement and frustration with the original release of Microsoft Word.

Word for Windows was a revelation. It was also a mess.

It was the first WYSIWYG (what you see is what you get) word processing program. (Amazing.) It would also occasionally-and-at-random crash your computer, erasing all of your work. (What.)

That was the tradeoff. You can use this program that simplifies a bunch of frustrating tasks. When it works well, its really great and it feels kinda like the future. But also you can’t trust the thing and occasionally there will just be a catastrophe. Caveat emptor.

(Sounds, yes, a bit like using an LLM for many workplace tasks.)

I’m sure there were cranky tech critics back then who insisted the technology was a piece of junk. Just fatally flawed. Who the hell is going to rely on a word processor for serious writing tasks when word processors randomly devour your work? And those cranky tech critics eventually looked foolish, because they were conflating a temporary problem with a systematic problem.

The thing about the bugs in early word processors and (I’m sure) VisiCalc is that users had good reason to be confident that the bugs would eventually get worked out. Sure, it’s clunky now. But that’s because these are early times. Give it five years and the software industry will work out the kinks. Microsoft’s entire business model is premised on releasing new versions of the product that make it more useful and reliable to the people actually using the product. There are billions of dollars at stake, in a marketplace whose incentives will reward the company that fixes the fatal “unrecoverable application errors.”

These are also early times for ChatGPT and other LLMs right now. But I think the comparison otherwise falls short. Because I’m not convinced the “bugs” we experience with LLMs are much like the bugs we experienced with early Word and VisiCalc.

I’m skeptical. I think its likely to be more akin to microtargeted advertising.

---

The thing I want to stress about microtargeted ads is that the current version is perpetually trash, and we’re always just a few years away from the bugs getting worked out.

This was true in the aughts. It was true in the teens. It is still true today.

I wrote about this example last year (in “Two failure modes of emerging technologies”), but let me recycle it: roughly once a month, Amazon.com sends me an email suggesting that I might want to buy either The MoveOn Effect or Analytic Activism. I am an especially bad target for these advertisements, because I am the author of both books.

Amazon has been in business for over twenty-five years. It has the largest, most sophisticated consumer database in the world. It is a $1.87 trillion company. And its online ad system still tries to sell authors their own books.

And it isn’t just Amazon. I am routinely served ads for products that I have already purchased. When we bought our house a few years ago, we purchased a whole-home water filter. We purchased it through the company’s website. And then I was constantly served programmatic ads for that same water filter for the next 6+ months. It cost a couple grand. It came with a warranty. There is no conceivable universe in which I’m likely to buy another one right away.

This was also a drum I kept banging back in the Cambridge Analytica days. Online advertising simply has never worked as well as its promoters claim. And we have, for decades, been caught in an endless hype-loop where critics warn that digital microtargeting is on the verge of becoming so good that it fractures democracy, even if the currently-existing products are full of errors. Give it five years and the online advertising industry will surely work out all the kinks! Except, five years later, the industry has grown but the underlying data is still crap.

There is an entire literature on this phenomenon. Maciej Ceglowski’s “the internet with a human face” is a favorite of mine, as is Jesse Frederick and Maurits Martijn’s “the new dotcom bubble is here: it’s called online advertising.” Or, if you’re up for a booklength treatment of the subject, there’s Tim Hwang’s Subprime Attention Crisis and Cathy O’Neil’s Weapons of Math Destruction and Virginia Eubanks’s Automating Inequality. (Those are just a few favorites that come immediately to mind. I could put together a whole syllabus on the topic.)

And the simple explanation of why the data underlying online advertising continues to be an ocean of garbage is that online advertising is a massive, barely-regulated industry. The money flows toward companies that can make the most compelling pitches to corporate executives, not to the companies whose products make the fewest errors.

That’s just how it is. The “bugs” in online advertising will never be corrected, because the marketplace neither demands nor rewards correction. There is enough money at stake that all the big actors have an interest in pretending everything works just fine already. …And, even if it doesn’t, we can always just focus on the imaginary product that will exist in five years.

---

So here’s the real question: is the marketplace for LLMs more like the marketplace for word processors or the marketplace for targeted advertisements?

I don’t know for certain. On the one hand, Sam Altman and all of his competitors recognize that AI hallucinations are a huge problem. They are indeed going to direct resources to solving it. On the other hand, as Colin Fraser suggested last month (in a barnburner essay that everyone ought to read), “Generative AI is a hammer and no one knows what is a nail.”

These may not be problems that transformer models can solve with more data. It may prove easier for the big players in the industry to just insist that the problem has been reduced, and/or it is no problem at all cometothinkofit.

That Pew survey showing 20% of employed Americans have tried ChatGPT for work gives us no insight as to whether they kept using it. We still don’t know what the revenue model of these phenomenally-expensive products will be, but I suspect it is not going to rest on individual end-users being so happy with a product that they decide to pay for it.

Ultimately, this brings me back to Ted Chiang’s essay from last year, “Will A.I. Become the New McKinsey?”

How we view the present and future of Artificial Intelligence probably turns on what we think about the current state of capitalism. Is A.I. going to inevitably improve (because markets)? Or is A.I. inevitably going to hollow out industries while providing shittier services (because markets)?

I lean toward the latter. I would like to be wrong! But I don’t think the nascent market structure of this industry is going to drive toward quality. I don’t expect the bugs to go away in five years. Not unless regulators step in, and withstand the intense pressure campaign they are already facing from tech billionaires.

So I don’t think ChatGPT is going to be much like VisiCalc. It’s just a lot more difficult for us to have and keep nice things under the market and regulatory constraints that have developed in the intervening decades.