Resources
News

A Tale of Two Webs: Cloudflare’s Pay per crawl aims to save content creators while other businesses race to make more content available

Chris Andrew Chris Andrew •

On July 1, Cloudflare, a leader powering nearly 20% of the internet, announced “pay per crawl”, a bold new approach that lets content creators charge AI crawlers for access to their work. It’s a major step toward helping a challenged category of publishers and original content creators gain more control and compensation as AI platforms reshape the web.

At Scrunch AI, we applaud Cloudflare’s leadership in helping to build a fairer, more sustainable internet. This is a pivotal moment for the future of the web and we’re excited to see these protections rolled out.

As AI discovery transforms how people find information, we see a different story starting to unfold as a result. There are two distinct webs emerging across the internet: content creators and publishers versus businesses with something to sell. Let’s break that down.

Web 1: Content creators and publishers

This web is made up of media outlets, bloggers, and creators whose work is their product. Their success hinges on producing original content (articles, analysis, reviews, guides) that others want to read, cite, and train on.

  • Estimated share of web traffic: ~20-30%*
  • Current challenges: Content fuels AI models, often without driving traffic or revenue in return.
  • Wants/needs: Protect intellectual property and secure fair compensation.
  • Implication of Pay per crawl: A vital tool to reclaim value. This lets creators monetize the extraction of their work by AI crawlers, helping offset traffic losses from AI Overviews and chatbots that answer users without sending them to the source.

Even within this cohort, new dynamics are emerging. A recent Wall Street Journal report surfaced that some Tier 1 publishers are seeing drops in organic search traffic as high as 20-60% due to AI Search. Where is the traffic going? It’s flowing to Tier 2 and Tier 3 publishers, smaller trade publications and niche outlets that AI models surface in response to user queries. Of note, many of the Tier 1 publishers have blocked AI crawlers from day 1, whereas Tier 2 players have allowed access to their content. It will be interesting to see how “Pay per crawl” impacts the landscape.

Web 2: Businesses with something to sell

This web is home to brands, retailers, and service providers; anyone who is trying to sell a product, service, or experience. These businesses invest in content (SEO, blogs, product pages) not to be paid for their words, but to drive discovery, visibility, and sales.

  • Estimated population size: ~50% of web traffic*
  • Current challenges: AI discovery is shifting traffic patterns; brands are often unclear how to stay visible, and recommended by AI.
  • Wants/needs: Ensure content is discovered, cited, and recommended by AI platforms to drive commercial results.
  • Implication of Pay per crawl: For these businesses, Pay per crawl is less about protecting content and more about staying visible. Charging AI crawlers could backfire, restricting the very exposure that drives conversions. Their priority isn’t to limit access, but to ensure AI assistants surface their offerings at key decision points.

The opportunity: These businesses need to double down on AI visibility strategies:

  • Create clear, unambiguous content that AI can parse and recommend.
  • Monitor where and how they appear in AI-generated results.
  • Optimize product descriptions and service pages for AI consumption.

Ultimately the primary consumer of content on the internet is moving to be AI Agents/Bots. A new web will emerge to support this.

Looking forward

This is a pivotal moment for how the internet evolves in an AI-first world. At Scrunch AI, we’ve seen firsthand the shift to our priority is to help brands stay visible, stay competitive, and thrive in this shifting landscape and we’re glad to be part of an ecosystem tackling these challenges together. We’re excited to see these solutions driving a fairer, more sustainable web.

*Sources: According to a combination of sources and data from Cloudflare and W3Techs. Content Creators (~20-30%), Businesses with something to sell (~50%), remaining balance of traffic is made up of academia, government, etc.