Cloudflare says: for every one AI reader, there are thousands of scraped pages
Cloudflare has calculated that for every living reader AI bots return to publishers, there are thousands of scraped pages. In the company's view, this is not the deal publishers agreed to.
For every living reader that artificial intelligence bots bring to publishers' websites, there are thousands of scraped pages. That is the estimate shared this week by Cloudflare, one of the largest internet infrastructure operators, through which a significant portion of global web traffic passes.
The number is unpleasant. And, more importantly, it breaks the original logic of the relationship between publishers and search engines.
What was the deal, and what went wrong?
The classic scheme looked simple. Publishers allow search crawlers to scan their pages. In return, search engines bring living people to websites — the very readers who can be monetized through advertising and subscriptions.
AI bots broke that arithmetic. They take content in enormous volumes, and in return give publishers a handful of readers. More precisely — one per thousands of "consumed" pages, if Cloudflare's calculations are to be believed.
This was discussed in a recent episode of the AdExchanger Talks podcast by Lara Cohen, Cloudflare's Vice President of Strategic Partnerships for Media, Authors and AI. Her wording deserves a direct quote: this is not what publishers agreed to when they allowed search engines to crawl their sites in exchange for traffic.
What does this change for publishers?
The practical question is this: if AI bots consume content on an industrial scale and return almost no traffic, who should pay for it, and how?
The very title of the discussion — "Making AI Pay Its Way" — sets the agenda. We are talking about a transition from the "content in exchange for traffic" model to one in which automated access to publishers' materials must have an explicit price.
Cloudflare's position here is no accident. Technically, the company sits between publishers and those who scrape their sites — it runs the infrastructure and therefore sees bot request volumes at the scale of the entire market. Data of this kind is hard to obtain from anywhere else: an individual publisher sees only their own statistics, while the aggregate picture is visible only to an operator of Cloudflare's scale.
For the Russian-speaking B2B market, the story has a direct reading. Russian media and content platforms face the same problem: training and operating generative models requires mass collection of texts, while compensation mechanisms that would take rightsholders' interests into account have not yet been built in the industry. The precedents being set now at the global level — including on the basis of Cloudflare's data — will define the framework for future agreements between publishers and AI developers.
In brief
Cloudflare has given the market a number around which negotiations can now be built: thousands of scraped pages per one reader. As long as publishers traded content for traffic, the exchange was at least symmetric. Now the symmetry is gone. What remains to be seen is what replaces it — money, licenses, or lawsuits. Judging by the AdExchanger Talks episode, the discussion is only beginning.
This article is based on an episode of the AdExchanger Talks podcast featuring Lara Cohen, Cloudflare's Vice President of Strategic Partnerships.
Source: AdExchanger
More from Артём Пасечник
157 articles
Related articles
- Elvis Presley Song Rights Holders Demand $41.1 Million from Anthropic
- Pinduoduo and Anthropic paid Kazakhstan 8 billion tenge in taxes
- Disney Will Show Super Bowl 2027 Through the Eyes of Star Wars Characters
- Mom's Touch to Open 50 Restaurants in Kazakhstan with Shin-Line
- Prada answered AI slop with a pencil: 6 million views and mice in the style of Disney