Starting Sept. 15, Cloudflare will block AI training and agent crawlers by default on ad-supported pages for new domains and free-tier customers, while continuing to allow search crawlers unless site owners change their settings.
The change was outlined by Cloudflare executive Chema Alonso at Media Party Barcelona and detailed in The Media Stack’s account of the talk. Mixed-use crawlers will be treated according to their most restrictive function, meaning a bot that handles both search and AI training could be blocked if training access is denied.
Alonso said automated traffic has overtaken human traffic on Cloudflare’s network for the first time. Cloudflare says more than 20% of the web sits behind its infrastructure, giving it a broad view of how search crawlers, AI agents and training bots move across publisher sites.
The economic problem is familiar. Search engines traditionally indexed publisher pages and sent readers back, where advertising or subscriptions could generate revenue. AI systems can use the same reporting to answer a question without producing the click that generates revenue.
Cloudflare says 52% of crawler requests on its network are now tied to AI training, while more than a third come from mixed-use bots. The company wants crawlers classified as Search, Agent or Training so publishers can allow, block or charge each category separately. The controls build on Cloudflare’s earlier move to block AI training crawlers by default.
Publishers have long used robots.txt files to tell automated crawlers which parts of a site they may access. Separately, the Robots Exclusion Protocol standardizes how those instructions are written and interpreted. It governs crawler access rules, not payment terms or licensing.
Cloudflare is also changing how it thinks publishers should be paid. Its Pay Per Crawl model compensated a publisher when a bot fetched a page. The company now says payment should instead happen when publisher content appears in an AI answer.
Two partners are already testing that approach. Ceramic.ai pays for non-paywalled news surfaced in its results, while You.com pays when premium paywalled content is used in a response.
Sponsored. Journalists, PR pros and communicators: the fall cohort of AI for Media starts October 13, six live Tuesday sessions with Pete Pachal plus two 1:1 coaching calls. Code AISEARCH500 takes $500 off the $1,500 price for anyone who found the course through AI search, a bigger discount than is offered anywhere else.
Cloudflare says transparency is central to the model. Its argument is that clearer information about who is taking content and why can create scarcity, and that scarcity can give publishers more leverage to strike licensing agreements. Media Copilot has separately reported on publishers using blocking tools in AI licensing negotiations.
Cloudflare counts more than 50 publisher-AI agreements since 2023 but acknowledges that licensing is unlikely to replace all of the referral and advertising revenue publishers have lost.
For newsrooms, the immediate change is more control over which machines can access their work. Product and audience teams will have to decide whether to allow agent access, block training or seek payment for some uses.
Those decisions are unfolding alongside other compensation experiments. Separately, Google is testing payments based on publisher content value across some of its AI products.
The hardest case for Cloudflare remains Googlebot, which can serve multiple purposes. Cloudflare wants blocking AI training to carry no search penalty, but Google has not agreed to that request.
After Sept. 15, the practical test will be whether mixed-use crawlers separate their functions, disclose them more clearly or simply lose access.
