Cloudflare just drew a hard line between search crawlers and AI bots. Starting September 15, 2026, the company will block mixed-use crawlers from ad-supported pages by default, and it’s replacing its Pay Per Crawl system with a new model that ties publisher payments to actual AI answers, not raw page fetches.

Key Takeaways

• Mixed-use AI crawlers blocked on ad pages starting September 15
• Pay Per Use replaces Pay Per Crawl to reward actual citations
• Ceramic.ai and You.com are the program’s first two AI partners
• Over 50% of AI crawl traffic re-fetches pages that never changed
• Bots now account for 57% of all web traffic, surpassing humans

Why Cloudflare Is Picking This Fight Now

For decades, the web ran on a simple deal. Search engines crawled your site, indexed it, and sent readers back. Those clicks funded ads, subscriptions, and sales. Everyone got something.

Generative AI broke that deal. AI answer engines now extract information, summarize it, and deliver responses without a click. The value flows one way. The publisher gets nothing.

The numbers tell the story. According to Cloudflare’s own data from 2025, Anthropic’s crawler accessed roughly 73,000 pages for every single referral it sent back to a publisher. OpenAI’s ratio stood at about 1,700 crawls per referral. Google, by comparison, ran at 14 to 1. AI companies are taking orders of magnitude more than they return.

Then came the bot traffic milestone. In June 2026, Cloudflare Radar data showed that automated systems accounted for roughly 57.4% of all HTTP requests to web content — surpassing human traffic for the first time. Cloudflare CEO Matthew Prince hadn’t expected this crossover until 2027. Agentic AI traffic pushed it ahead of schedule.

That context made the timing urgent. Bots are now the majority of web traffic, AI crawlers return almost no referral value, and publishers are bleeding revenue. According to Forbes, some publishers have lost 20 to 90 percent of their traffic and revenue over the past year. Cloudflare, which manages traffic for roughly 20 percent of the web, decided that waiting was no longer an option.

How The New Default Settings Work

The policy centers on one core change: separating search crawling from AI use. Beginning September 15, 2026, Cloudflare’s default settings will allow traditional search bots while blocking AI training and agent crawlers on pages that display ads.

The new defaults apply to three groups: new Cloudflare customers, new sites created by existing customers, and all current free-tier users who haven’t changed their settings. Existing paid customers who already configured their preferences won’t be affected unless they choose to opt in.

Cloudflare is splitting AI traffic into three categories that publishers can manage independently:

• Search: indexes content for later retrieval in search results
• Agent: accesses a site on behalf of a user in real time
• Training: collects content to train or fine-tune AI models

The real target is what Cloudflare calls “mixed-use crawlers” — bots that blend search, agent, and training activity into a single crawler without letting publishers choose between those uses. Apple, Google, and Microsoft’s Bing all operate crawlers that could fall into this category, though each offers some form of AI opt-out. Cloudflare is pushing all of them to split their bots by purpose. If they don’t, ad-supported pages will block them by default.

The message is straightforward: publishers shouldn’t have to choose between being discoverable in search and giving away their work to train AI models for free.

Pay Per Use And The First AI Partners

Cloudflare launched its original Pay Per Crawl system in private beta on July 1, 2025. It let publishers charge AI companies a fee each time a crawler fetched a page, using HTTP 402 “Payment Required” responses. It was a start, but Cloudflare now says crawling is a poor measure of value. A single page might get crawled once and cited in thousands of AI answers, or crawled repeatedly and never cited at all.

The replacement is Pay Per Use — a model where publishers get paid when their content actually appears in an AI-generated result, not just when it gets fetched. Cloudflare’s data backs the shift: more than 50% of AI crawl traffic goes toward re-fetching pages that haven’t even changed. That’s wasted bandwidth for publishers and wasted compute for AI companies.

Two partners are piloting the model. Ceramic.ai, an API-based search company founded by Anna Patterson, runs a pay-per-query system. When a publisher opts in and their content surfaces in Ceramic’s AI search results, they get paid. Publishers also gain reporting on which queries triggered their content, the exact snippets displayed, and their ranking position — data Cloudflare describes as useful for answer engine optimization.

“By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results,” said Patterson.

You.com takes a different approach. Its model lets AI agents pay on demand for a specific piece of premium content at the moment it’s needed, with no upfront commitment from either side. Cloudflare says other AI companies can customize the framework for their own use cases.

A new Attribution Business Insights dashboard rounds out the package, showing publishers how AI bots access their content, where that content is cited, and how much human traffic each AI platform sends back.

What This Means For Publishers And AI Companies

For large publishers with heavy traffic and ad-driven business models, Cloudflare’s changes offer real leverage. They can now block AI training by default, charge for access, and track exactly how their content gets used. The tools are free for all Cloudflare customers, including those on the free tier.

But the picture is less clear for smaller sites. As MediaCopilot noted, Cloudflare’s default blocking policy is tied to pages with advertising, and its compensation plans focus on commercial use cases. Independent bloggers and hobby sites get the same on-off switches, but switches don’t equal leverage. There’s still no obvious path for small creators to get paid when their work trains a model and never comes back with a citation.

For AI companies, the stakes are operational. Pipelines built on bulk crawling will need opt-in checks, credentials, or paid access workflows. The cost of sourcing web-scale training data just became more explicit. Companies like Google face a particular squeeze — Cloudflare pointed out that the largest search engine currently holds roughly twice as much data as leading AI companies because publishers feel forced to allow AI access just to stay visible in search results.

There’s also the intermediary question. Cloudflare would potentially control crawler identification, the permissions layer, usage measurement, and the payment infrastructure. Publishers may gain leverage over AI companies while growing more dependent on a single gatekeeper.

The Bigger Picture For The Open Web

Cloudflare is not alone in this space. TollBit, ProRata, and Microsoft have all moved into content licensing, each with a different theory of who sets the terms. Really Simple Licensing is pushing for an open standard before any single company locks in the rules. According to Forbes, nearly 70 percent of publishers expect AI licensing deals to generate at least some revenue over the next three years, though most see it as a minor source today.

What Cloudflare is building goes beyond a single feature. The company is framing its vision around what it calls the “Agentic Internet” — a web where autonomous agents are the primary visitors, and the economic rules need to match. Permission, attribution, and payment become the foundation, not afterthoughts.

The unsolved problem is attribution itself. An AI-generated answer might blend dozens of sources, paraphrase an original idea, or use reporting without displaying a citation. How the value of each source gets calculated, who audits the results, and how disputes get resolved — none of these questions have clean answers yet.

Still, Cloudflare’s move marks a structural shift. For the first time, a company managing roughly a fifth of all web traffic is changing the default from open access to blocked, and building payment rails that tie compensation to outcomes rather than raw data extraction. Whether the model scales or fractures into competing walled gardens will depend on how many AI companies and publishers actually opt in — and whether the attribution math ever gets good enough to make everyone trust it.

Digital Trendings is your trusted source for AI news and updates — stay tuned for more.