The crawl is coming from inside the house

On the one hand, my cup runneth over with browsers that are putting the “agent” back in user agent: Comet. Dia. Project Mariner. Untitled OpenAI browser project.

On the other hand, the backlash against AI crawlers from folks who make a living publishing content on the internet continues apace, and has found a platform champion in Cloudflare.

What do these two things have in common? The narrative is about the struggle between platforms (the EU might call them gatekeepers): if you want to be a major technology platform, it’s an existential risk not to have a browser (or maybe a phone.) The more screen time the platform owns, the more powerful they are.

There is truth to this, but I think it undersells a more powerful force: changing consumer preference.

If you’re not a big technology platform — and maybe if you are — there’s a much harder question than whether OpenAI supplants Google as the dominant platform:

Regardless of what gatekeepers want: do people want to browse the web any more?

In my view, the answer is no. There are still things that are experiential on the web: fashion, art, reading for pleasure on a handful of trusted publishers. But for many the modern web browsing experience has become mostly a chore.

Technology evolves, and people don’t have to browse any more. So they mostly won’t.

Today most people access AI tools through platforms that look an awful lot like the platforms of yesterday: spartan web pages with big text boxes in the middle of them. But there’s no reason this Google-circa-2004 model has to be dominant form factor. An AI browser is just another mechanism to connect AI with the existing web — one that runs on millions of consumer devices, from residential networks, instead of from a Kubernetes cluster in Azure. The browser form factor itself may be mostly vestigial.

For publishers, asking for compensation from AI labs that train on their data or platform aggregators that remix their content seems like fair play. As a society, we need to decide how we value human writing and how to pay for it.

From a mechanism perspective, though, the genie is out of the bottle. War can be waged against internet gatekeepers, and perhaps publishers can keep AI platforms from accruing value at their expense. But publishers can’t keep that value themselves if it’s dependent on folks continuing to browse.