---
title: "Cloudflare will block AI training crawlers by default on ad-supported sites"
id: "8833"
type: "post"
slug: "cloudflare-ai-training-crawlers-default-block"
published_at: "2026-07-01T17:36:10+00:00"
modified_at: "2026-10-28T11:55:51+00:00"
url: "https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block/"
markdown_url: "https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block.md"
excerpt: "The company says new controls will let publishers separate search, agent use and model training"
taxonomy_category:
  - "News"
taxonomy_post_tag:
  - "Cloudflare"
  - "webscraping"
  - "AI traffic"
  - "AEO"
---

# Cloudflare will block AI training crawlers by default on ad-supported sites

The company says new controls will let publishers separate search, agent use and model training

![Exterior of Cloudflare's corporate headquarters](https://mediacopilot.ai/wp-content/uploads/2026/07/AdobeStock_301259118_Editorial_Use_Only-400x225.jpeg)

Cloudflare says that starting Sept. 15 it will block AI training and agent crawlers by default on ad-supported pages, while continuing to allow search indexing unless site owners choose otherwise. (Credit: Sundry Photography - stock.adobe.com)

Jul 1, 2026

By       [Romy Abu-Fadel](https://mediacopilot.ai/author/romy/)

Cloudflare [said](https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/)
 Wednesday it will begin blocking AI training and agent crawlers by default on ad-supported websites, a change that could force companies such as Google, Apple and Microsoft to more clearly separate search indexing from AI training if they want continued access to large parts of the web.

The policy, scheduled to take effect Sept. 15, applies to new Cloudflare customers, new sites added by existing customers and existing Free-tier customers who have not changed their settings. Search crawlers will remain allowed by default, but training and agent crawlers will be blocked on pages that display ads.

The company said the changes are designed to help publishers remain visible in AI-powered search results while preventing their content from being used for AI training or autonomous agents without permission or compensation.

“Now that the majority of traffic is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Matthew Prince, Cloudflare’s co-founder and CEO.

[Are you an AI Agent? Read more here >](https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block.md)

### Splitting up mixed use crawlers

The Web giant said bots that combine search, AI training and agent activity—known as mixed use crawlers—without letting site owners choose among those uses will be blocked on ad-supported pages when training or agent access is blocked. In a company [blog post](https://blog.cloudflare.com/content-independence-day-ai-options/)
, Cloudflare named [Googlebot, Applebot and BingBot](https://blog.cloudflare.com/content-independence-day-ai-options/)
 as multi-purpose crawlers that could be affected by the most restrictive applicable rules.

“We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training,” Prince said.

Cloudflare said customers will be able to manage three categories of AI traffic: Search, which indexes content for later retrieval; Agent, which accesses a site on behalf of a user in real time; and Training, which collects content to train or fine-tune models. The controls are available to all Cloudflare customers, including those on the Free tier.

That distinction matters for smaller sites. A spokesperson for Cloudflare said the new controls are intended to give all website owners more options for managing AI traffic, not only publishers with ads or subscriptions. But the default blocking policy is tied to pages with advertising, and Cloudflare’s compensation plans remain focused on commercial use cases where AI systems access or surface publisher content.

Alongside the new crawler controls, Cloudflare is expanding analytics to show publishers how bots interact with their content and how much traffic AI platforms send back. The company is also pushing into what it calls Answer Engine Optimization, or AEO, offering tools it says will help customers understand how often their content is cited or surfaced in AI-generated answers.

Cloudflare also announced efforts to reduce unnecessary AI crawling. According to the company, more than half of AI crawler traffic is spent repeatedly checking web pages that have not changed. Because Cloudflare sits between websites and online traffic, it says it can signal to AI companies when pages have been updated and worth revisiting. The company said it is testing those signals with AI firms and plans a broader rollout later this year.

### New compensation model

The company is also expanding its publisher compensation strategy by evolving its Pay Per Crawl program into a new system called Pay Per Use. Rather than paying publishers when content is crawled, the new model is designed to compensate them when their content is actually used in AI products. Cloudflare said it is working with AI companies including [Ceramic.ai](http://ceramic.ai/)
 and [You.com](http://you.com/)
 on the initiative. Under the arrangements, publishers could be paid when their content appears in AI search results or when AI agents access premium content on demand.

But the model does not yet answer the hardest compensation question: what happens when a publisher’s work is used for model training but never appears in a cited answer? Asked whether Pay Per Use compensates publishers in that scenario, The spokesperson said the program is aimed at “programmatic, real-time access and discovery,” and described Pay Per Crawl and Pay Per Use as only two possible economic frameworks.

“The digital landscape is evolving rapidly,” said Marrissa Holloway for Cloudflare. “We welcome ideas from publishers, creators, and AI companies alike on how to build a thriving agentic Internet.”

Holloway did not directly say what Cloudflare’s cut of any revenue generated would be. “It has always been our philosophy that our customers derive many multiples of value more than they pay us,” she said.

- Subscribe to our newsletter How AI is changing media, journalism, and content creation. [Learn More](https://mediacopilot.ai/newsletter/)

![Image](https://mediacopilot.ai/wp-content/uploads/2025/05/ads-1024x682.jpg)

### The Media Copilot’s take

Cloudflare is not solving AI compensation for the whole Web. It’s building a bargaining layer for larger publishers with enough traffic and revenue to measure, block and negotiate. That helps the larger content outlets, but smaller sites and independent publishers will get switches to turn on and off. That’s useful, but switches don’t mean they have leverage. The long tail of the Web—the indy blog sites, community web pages and hobby sites—can say “no” more clearly, but there still no obvious way for them to get paid when their work is used for an AI’s training data and never comes back with a citation or link.

Contributors

- [Romy Abu-Fadel](https://mediacopilot.ai/author/romy/) : AuthorRomy Abu-Fadel is a journalist, researcher, and 2026 graduate of Georgetown University's Edmund A. Walsh School of Foreign Service. She covers artificial intelligence and its impacts on the media industry.

  - [Christopher Allbritton](https://mediacopilot.ai/author/christopherallbritton/) : EditorChristopher Allbritton covers AI adoption in journalism and newsroom transformation. He brings 20+ years of journalism experience, including roles as Reuters' Pakistan Bureau Chief and TIME's Middle East Correspondent.

Category: [News](https://mediacopilot.ai/category/news/)
Tags:[Cloudflare](https://mediacopilot.ai/tag/cloudflare/)
| [webscraping](https://mediacopilot.ai/tag/webscraping/)
| [AI traffic](https://mediacopilot.ai/tag/ai-traffic/)
| [AEO](https://mediacopilot.ai/tag/aeo/)

Share this post:

[Facebook](https://www.facebook.com/sharer/sharer.php?u=https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block/&display=popup&ref=plugin&src=share_button)
[Tweet](https://twitter.com/share?url=https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block/&text=Cloudflare%20will%20block%20AI%20training%20crawlers%20by%20default%20on%20ad-supported%20sites)
[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https://mediacopilot.ai/cloudflare-ai-training-crawlers-default-block/)
[Email](/cdn-cgi/l/email-protection#714e0204131b1412054c281e0403544341170318141f15544341191002544341021910031415544341101f54434110030518121d1454434106180519544341081e045f57524142494a131e15084c321d1e0415171d10031454434106181d1d544341131d1e121a5443413038544341050310181f181f16544341120310061d140302544341130854434115141710041d055443411e1f54434110155c020401011e03051415544341021805140254413554413019050501025442305443375443371c14151810121e01181d1e055f1018544337121d1e0415171d1003145c10185c050310181f181f165c120310061d1403025c15141710041d055c131d1e121a544337544135544130)