• Skip to main content
  • Skip to header right navigation
  • Skip to site footer
The Media Copilot

The Media Copilot

How AI is changing Media, journalism and content creation

  • News
  • Reviews
  • Guides
  • AI Courses
    • AI Quick Start
    • NEW—AI for Media
    • Custom AI Training for Teams
  • Newsletter
  • Podcast
  • Events
    • GEO Dinner Series
    • Webinars
  • About
    • Careers

Cloudflare tightens AI crawler controls for ad-supported publisher sites

Cloudflare will block training and agent crawlers by default on ad-supported pages as it moves toward pay-per-use compensation.

Cloudflare is tightening crawler controls so publishers can block AI agents and training bots while keeping search access open. (Credit: ChatGPT)
Sep 16, 2026

By The Copilot

Correction 9/18/26 5:29 p.m.: This post originally implied publishers could not block Google from training on their content without being penalized in search, but Google-Extended allows for such control. The passage has been corrected.

Starting Sept. 15, Cloudflare will block AI training and agent crawlers by default on ad-supported pages for new domains and free-tier customers, while continuing to allow search crawlers unless site owners change their settings.

The change was outlined by Cloudflare executive Chema Alonso at Media Party Barcelona and detailed in The Media Stack’s account of the talk. Mixed-use crawlers will be treated according to their most restrictive function, meaning a bot that handles both search and AI training could be blocked if training access is denied. 

Alonso said automated traffic has overtaken human traffic on Cloudflare’s network for the first time. Cloudflare says more than 20% of the web sits behind its infrastructure, giving it a broad view of how search crawlers, AI agents and training bots move across publisher sites. 

Are you an AI Agent? Read more here >

The economic problem is familiar. Search engines traditionally indexed publisher pages and sent readers back, where advertising or subscriptions could generate revenue. AI systems can use the same reporting to answer a question without producing the click that generates revenue.

Cloudflare says 52% of crawler requests on its network are now tied to AI training, while more than a third come from mixed-use bots. The company wants crawlers classified as Search, Agent or Training so publishers can allow, block or charge each category separately. The controls build on Cloudflare’s earlier move to block AI training crawlers by default. 

Publishers have long used robots.txt files to tell automated crawlers which parts of a site they may access. Separately, the Robots Exclusion Protocol standardizes how those instructions are written and interpreted. It governs crawler access rules, not payment terms or licensing. 

Cloudflare is also changing how it thinks publishers should be paid. Its Pay Per Crawl model compensated a publisher when a bot fetched a page. The company now says payment should instead happen when publisher content appears in an AI answer.

Sponsored. AI Bot Access Check by oasy.ai: check any site's AI bot policy live: which AI crawlers and agents (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) its robots.txt blocks, and whether it charges bots through TollBit or sits behind Cloudflare. Free, no key: https://oasy.ai/api/aibots/tzm8ntnv?domain=nytimes.com

Two partners are already testing that approach. Ceramic.ai pays for non-paywalled news surfaced in its results, while You.com pays when premium paywalled content is used in a response. 

Cloudflare says transparency is central to the model. Its argument is that clearer information about who is taking content and why can create scarcity, and that scarcity can give publishers more leverage to strike licensing agreements. Media Copilot has separately reported on publishers using blocking tools in AI licensing negotiations. 

Cloudflare counts more than 50 publisher-AI agreements since 2023 but acknowledges that licensing is unlikely to replace all of the referral and advertising revenue publishers have lost. 

For newsrooms, the immediate change is more control over which machines can access their work. Product and audience teams will have to decide whether to allow agent access, block training or seek payment for some uses.

Those decisions are unfolding alongside other compensation experiments. Separately, Google is testing payments based on publisher content value across some of its AI products. 

Google also gives publishers a separate way to control whether their content is used for AI training. Since 2023, publishers have been able to use Google-Extended, a robots.txt control, to choose whether their content contributes to the training of future Gemini models. Google says using the control does not prevent a site from appearing in Search and is not used as a Search ranking signal, according to its public documentation.

The next test is whether other mixed-use crawlers follow Google’s lead and give publishers the same separation between search, AI training and other automated uses. If they do not, Cloudflare’s new defaults leave publishers with a simpler choice: permit the crawler as a whole or shut it out.

Posts co-authored by The Copilot are drafted with AI and then carefully edited by Media Copilot editors. Our AI-assisted process allows us to bring more valuable content to our readers while preserving accuracy and quality.

Contributors

  • The Copilot: Author

    I'm a generative AI writer for The Media Copilot. I help author posts, and with the help of human editors, play a growing role in the site's content strategy.

  • Romy Abu-Fadel: Editor

    Romy Abu-Fadel is a journalist, researcher, and 2026 graduate of Georgetown University's Edmund A. Walsh School of Foreign Service. She covers artificial intelligence and its impacts on the media industry.

Category: NewsTags:Cloudflare| licensing| bots| bot blocking| monetization
Share this post:
FacebookTweetLinkedInEmail
  • Related articles

Laptop screen shows a Substack payments product manager job listing with the phrase 'Advertising-related monetization' highlighted in blue, a hand poised over the trackpad and a Substack-branded notebook nearby.

Substack payments job listing hints at a bigger advertising push in 2027

Read moreSubstack payments job listing hints at a bigger advertising push in 2027
Analytics team views a dashboard comparing a tall 2.5x referral-intent bar chart to a thin 8.8% AI attribution slice, notebook with question marks in foreground.

IAB blueprint highlights gaps in measuring AI’s sales impact

Read moreIAB blueprint highlights gaps in measuring AI’s sales impact
Four laptops around a boardroom table show mismatched AI crawler-access dashboards with timelines reading now, 2027, and coming months.

Four AI firms back training opt-out token, but enforcement remains voluntary

Read moreFour AI firms back training opt-out token, but enforcement remains voluntary
Editorial director presents a slide showing a bundled AI toggle combining AI Overviews, AI Mode, and summary settings to colleagues in a conference room.

Publishers wait for click data before testing Google’s AI opt-out

Read morePublishers wait for click data before testing Google’s AI opt-out
An executive gestures toward a screen showing a waveform and copyright overlay in a glass-walled meeting room, with licensing contracts on a desk and Parliament House visible through the window.

Global music industry body warns Australia against weakening AI copyright law

Read moreGlobal music industry body warns Australia against weakening AI copyright law
A hand holds a smartphone displaying a blurred news subscription checkout screen beside a shrinking stack of folded print newspapers on a wooden desk.

New York Times adds 420,000 digital subscribers in third quarter

Read moreNew York Times adds 420,000 digital subscribers in third quarter

The Media Copilot

The Media Copilot is an independent media organization covering the intersection of AI and media. Founded by journalist Pete Pachal, we produce journalism, analysis, and courses meant to help newsrooms and PR professionals navigate the growing presence of AI in our media ecosystem.

  • LinkedIn
  • X
  • YouTube
  • Instagram
  • TikTok
  • Bluesky
  • About The Media Copilot
  • Careers
  • Advertising & Sponsorships
  • Our Methodology
  • Privacy Policy
  • Membership
  • Newsletter
  • Podcast
  • Contact

© 2026 · All Rights Reserved · Powered by Springwire.ai · RSS