A year ago, Cloudflare drew a line in the sand against unbridled AI crawling of the internet. Exactly one year later (again on Canada Day) the company took what it says is the next major step on that journey, introducing new tools for publishers and content creators to not just block bots from crawling their content, but charge them for access.

To me, the most interesting part of this is the new Pay Per Use framework. This builds on the existing Pay Per Crawl system, which charged bots whenever they crawled a page. But that straightforward approach didn’t necessarily capture the value of the crawl—once captured by an AI crawler, a piece of content could be used multiple times, in hundreds or even thousands of answers. On the other hand, something could be crawled and never used at all.

Pay Per Use fixes this by compensating the content owner when their content is actually used in an AI answer. Theoretically, if you published something unique, valuable, and optimized for machines to read, it could end up paying dividends for as long as people ask about it. And knowing that is part of the new system, too—Cloudflare promises analytics for content owners so they know how their content is being used. It’s also going to have a better system of telling bots when content hasn’t been updated so they don’t keep re-crawling the same static page over and over.

The system sounds like a sensible evolution to Pay Per Crawl—at least for inference (i.e. AI search engines). For AI training bots, Pay Per Crawl actually strikes me as the better solution since it’s more “one and done.” And how would you measure the value of an individual piece of content in a training set anyway?

All of this depends on a workable payment system, of course, and Cloudflare shared details on how it’s evolving that part of the framework. The new Monetization Gateway is straightforward: 

• a bot tries to access content
• the gateway responds with the payment needed and how to pay
• the bot deposits the payment and gets a proof of payment
• The bot then re-requests the content with the proof
• the gateway checks it and bestows access.

It’s all nice in theory, but this kind of usage-based pricing becomes a bookkeeping nightmare on the content owner’s side. This is one of the big reasons micropayments never took off in digital publishing—the revenue from a small payment by a single customer was never worth the processing hassle.

Cloudflare says its unique position as a content delivery network helps solve these problems. It’s already tracking and classifying the bots, so it’s easy to add the payment credential to the process. There’s no “account creation” or anything like that—the bot just shows the receipt. And it’s all done on an open protocol, with no checkout pages or separate payment API. Apparently, there are advantages to managing traffic for 20% of the web.

Cloudflare is refreshingly honest that its new Pay Per Use system is an experiment. How this all plays out depends largely on adoption, not just by publishers but also by AI companies and data brokers. Lots of people often say that digital publishing needs its Napster moment—when the music industry transitioned from sketchy Napster downloads to the “legit” option of iTunes. But iTunes downloads were aimed at individuals. Nobody typing a search into Google or Claude is deciding what content to pay for. This is all determined at the company level, and companies will always choose to get the best/most data for the least cost.

And that will ultimately come down to a simple equation: Is it less costly to get the data they want via Cloudflare’s system? If it’s not, it will remain merely an experiment.

The post Cloudflare’s new plan could change how AI pays publishers appeared first on The Media Copilot.

Cloudflare said Wednesday it will begin blocking AI training and agent crawlers by default on ad-supported websites, a change that could force companies such as Google, Apple and Microsoft to more clearly separate search indexing from AI training if they want continued access to large parts of the web.

The policy, scheduled to take effect Sept. 15, applies to new Cloudflare customers, new sites added by existing customers and existing Free-tier customers who have not changed their settings. Search crawlers will remain allowed by default, but training and agent crawlers will be blocked on pages that display ads.

The company said the changes are designed to help publishers remain visible in AI-powered search results while preventing their content from being used for AI training or autonomous agents without permission or compensation. 

“Now that the majority of traffic is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Matthew Prince, Cloudflare’s co-founder and CEO. 

Splitting up mixed use crawlers

The Web giant said bots that combine search, AI training and agent activity—known as mixed use crawlers—without letting site owners choose among those uses will be blocked on ad-supported pages when training or agent access is blocked. In a company blog post, Cloudflare named Googlebot, Applebot and BingBot as multi-purpose crawlers that could be affected by the most restrictive applicable rules.

“We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training,” Prince said. 

Cloudflare said customers will be able to manage three categories of AI traffic: Search, which indexes content for later retrieval; Agent, which accesses a site on behalf of a user in real time; and Training, which collects content to train or fine-tune models. The controls are available to all Cloudflare customers, including those on the Free tier.

That distinction matters for smaller sites. A spokesperson for  Cloudflare said the new controls are intended to give all website owners more options for managing AI traffic, not only publishers with ads or subscriptions. But the default blocking policy is tied to pages with advertising, and Cloudflare’s compensation plans remain focused on commercial use cases where AI systems access or surface publisher content.

Alongside the new crawler controls, Cloudflare is expanding analytics to show publishers how bots interact with their content and how much traffic AI platforms send back. The company is also pushing into what it calls Answer Engine Optimization, or AEO, offering tools it says will help customers understand how often their content is cited or surfaced in AI-generated answers.

Cloudflare also announced efforts to reduce unnecessary AI crawling. According to the company, more than half of AI crawler traffic is spent repeatedly checking web pages that have not changed. Because Cloudflare sits between websites and online traffic, it says it can signal to AI companies when pages have been updated and worth revisiting. The company said it is testing those signals with AI firms and plans a broader rollout later this year. 

New compensation model

The company is also expanding its publisher compensation strategy by evolving its Pay Per Crawl program into a new system called Pay Per Use. Rather than paying publishers when content is crawled, the new model is designed to compensate them when their content is actually used in AI products. Cloudflare said it is working with AI companies including Ceramic.ai and You.com on the initiative. Under the arrangements, publishers could be paid when their content appears in AI search results or when AI agents access premium content on demand. 

But the model does not yet answer the hardest compensation question: what happens when a publisher’s work is used for model training but never appears in a cited answer? Asked whether Pay Per Use compensates publishers in that scenario, The spokesperson said the program is aimed at “programmatic, real-time access and discovery,” and described Pay Per Crawl and Pay Per Use as only two possible economic frameworks.

“The digital landscape is evolving rapidly,” said Marrissa Holloway for Cloudflare. “We welcome ideas from publishers, creators, and AI companies alike on how to build a thriving agentic Internet.”

Holloway did not directly say what Cloudflare’s cut of any revenue generated would be. “It has always been our philosophy that our customers derive many multiples of value more than they pay us,” she said.

The Media Copilot’s take

Cloudflare is not solving AI compensation for the whole Web. It’s building a bargaining layer for larger publishers with enough traffic and revenue to measure, block and negotiate. That helps the larger content outlets, but smaller sites and independent publishers will get switches to turn on and off. That’s useful, but switches don’t mean they have leverage. The long tail of the Web—the indy blog sites, community web pages and hobby sites—can say “no” more clearly, but there still no obvious way for them to get paid when their work is used for an AI’s training data and never comes back with a citation or link.

The post Cloudflare will block AI training crawlers by default on ad-supported sites appeared first on The Media Copilot.

Cloudflare, Inc. and beehiiv, a newsletter-focused publishing platform, announced a strategic partnership on June 23, which they say will give independent creators greater control over how AI tools access, discover, and use their work.

Cloudflare’s advanced Crawl Control technology is integrated into the beehiiv platform, offering tools to monitor AI crawler activity in real time and manage those crawlers’ access to their content. This integration gives creators two options for managing their digital footprint: publishers can either allow all or some AI search engines to freely crawl their content for greater visibility or completely block AI scraping to protect their archives for future monetization and licensing.

Managing AI bots historically required manually updating robots.txt files or configuring firewalls. The Cloudflare–beehiiv partnership removes these steps, allowing publishers of all sizes, from major media organizations to independent creators, to easily set automated preferences through the platform’s standard dashboard settings. 

Key features of the integration include personalized analytics showing exactly which AI crawlers are accessing their content, which are being blocked, and how much referral traffic they generate, as well as one-click controls to allow or block specific AI models and automatic updates that extend those controls to new crawlers as they appear. 

Tyler Denk, co-founder and CEO of beehiiv, said AI bots from OpenAI, Anthropic, Google, and Microsoft had crawled content hosted on the platform 490 million times. 

“Creators and publishers should own and control their work,” Denk wrote, adding that they “may want to be indexed to maximize AI discovery” or “may want to keep [their] valuable content private and not crawled by AI bots.” 

Matthew Prince, co-founder and CEO of Cloudflare, echoed Denk’s statement: “As the internet evolves, [Cloudflare’s] commitment remains the same: ensuring creators have the tools they need to thrive,” Price said. “This partnership with beehiiv is the next logical step in that mission, giving newsletter operators the transparency and control to navigate the AI era on their own terms.”

AI Crawl Control has now launched in beta for all beehiiv users, giving publishers new visibility into  how AI systems access their content and the traffic those crawlers generate.  

The post Creators get new say over AI scraping through Cloudflare–beehiiv partnership  appeared first on The Media Copilot.

For the first time in the internet’s history, bots account for more web traffic than humans.

Cloudflare CEO Matthew Prince announced the milestone this week, according to Tom’s Hardware, noting that automated traffic has now eclipsed human-generated requests online, months ahead of even his own projections.

“Welp, that happened faster than I predicted,” Prince wrote on X. “Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet’s history.”

According to Cloudflare’s Radar data, bots represented roughly 57% of all HTTP requests as of late April 2026, with humans accounting for the remaining 43%. Bot traffic has held between 53% and 60% in the weeks since. Prince said the actual crossover occurred in the last few months, though the data is messy enough that pinning down an exact date is difficult.

The shift underscores how quickly AI agents have transformed web traffic patterns. Before the generative AI era, bot traffic sat at around 20% of all web activity, with Google’s web crawler serving as the largest single source. Now, AI agents performing tasks on behalf of users are generating requests at a scale that dwarfs human browsing behavior.

Prince illustrated the contrast at SXSW earlier this year: “If a human were doing a task—let’s say you were shopping for a digital camera—you might go to five websites. Your agent or the bot that’s doing that will often go to 1,000 times the number of sites that an actual human would visit. So it might go to 5,000 sites. And that’s real traffic, and that’s real load, which everyone is having to deal with and take into account.”

The reaction to Prince’s announcement was swift. Tech billionaire Elon Musk replied with a single “Wow” to the post.

The full picture is more nuanced. While bots now dominate HTML request traffic—reading pages, scraping content, indexing sites—humans still account for roughly 65% of total web activity when the metric expands to include app usage, video streaming, maps, and social media scrolling. Bots have overtaken humans in the specific act of navigating and reading the web, but not in the broader measure of people actually using the internet.

Cloudflare, which handles approximately one-fifth of all global web traffic, has been tracking the trend closely. The company’s 2026 Threat Intelligence Report also found that bots now account for 94% of all login attempts across its network, meaning only 6% of login attempts come from actual humans.

The crossing point Prince initially forecast for 2027 arrived in 2026. What once required a two-year runway happened in a matter of months.

The post Cloudflare CEO: Bots have overtaken human traffic online appeared first on The Media Copilot.

The same big tech companies that are stripping news publishers of site traffic are now dictating the terms of the emerging AI licensing market, and taking a significant cut in the process, according to a new report from the Open Markets Institute.

The report, “Same Gatekeepers, New Tollbooths: Mapping the AI Content Licensing Market,” argues that publishers are trapped in what the authors call a “double bind.” As Big Tech develops commercial AI products that siphon readers away from news sites, those same companies are positioning themselves as the gatekeepers of whatever alternative revenue streams emerge.

“Big Tech is occupying both sides of the value chain simultaneously,” write the authors, Courtney Radsch and Karina Montoya of the institute’s Center for Media & Digital Governance. They warn that “the deal structures, price precedents, intermediary take rates, and governance norms taking shape now will be difficult to revise once they are normalized.”

The report examines a growing ecosystem of AI content licensing marketplaces. Some are independent startups. Others are built by the very companies publishers are trying to negotiate with:

• ScalePost takes roughly 15% of revenue earned by rights holders.
• Cloudflare, which handles about 20% of global web traffic, takes an estimated 30% cut through its pay-per-crawl marketplace.
• ProRata.ai, which operates an answer engine built exclusively on licensed content, splits subscription and advertising revenue 50/50 with publishers. More than 500 publishers had signed up as of last summer.
• TollBit and Sphere.ai allow publishers to retain 100% of their revenue, instead charging AI companies a separate transaction fee.
• Microsoft’s Publisher Content Marketplace, announced in February, follows a pay-per-use model — but it’s not yet clear how much Microsoft will keep.

The report points to Spotify as a benchmark: despite a 30% take rate, the streaming model allowed music rights holders to earn significant revenue and stabilize the industry during a turbulent transition. The authors argue similar scrutiny is needed for AI licensing marketplaces, particularly when Big Tech is building the scaffolding.

“Regulatory attention is warranted on these platform operators in order to mitigate their data access advantages and ability to set de facto and potentially coercive standards for an industry in which no independent standards yet exist,” the authors write.

The post New report finds wide disparity in AI tollbooths for publishers appeared first on The Media Copilot.

Cloudflare today launched a feature that automatically converts HTML pages to markdown when AI agents request them, potentially changing how newsrooms publish content for AI-powered search and discovery.

The feature, called Markdown for Agents, uses content negotiation headers to detect when an AI system requests a page. When detected, Cloudflare’s network converts the HTML to markdown on the fly before serving it.

This matters because AI systems increasingly drive traffic to news sites, and those systems prefer markdown over HTML. The conversion happens automatically at the network edge, removing the need for publishers to maintain separate markdown versions of their content.

How it works

AI agents add an Accept: text/markdown header to their requests. Cloudflare detects this, fetches the original HTML from the origin server and converts it to markdown before returning it to the agent.

The company provides a simple curl example:

curl https://developers.cloudflare.com/page \
  -H "Accept: text/markdown"

Cloudflare includes an x-markdown-tokens header in responses that estimates token count, letting developers calculate context window sizes or plan chunking strategies.

Why markdown for AI

Markdown’s explicit structure makes it easier for AI systems to process. HTML pages have grown heavier over the years, making them harder to parse. AI agents typically filter out non-essential elements and scan for relevant content — a process markdown simplifies.

Most AI pipelines already convert HTML to markdown as a standard step, but this wastes computation, adds costs and processing complexity, and may not reflect how content creators intended their material to be used.

Cloudflare’s approach moves that conversion to the network edge, where it can happen efficiently at scale.

Content signals included

Converted responses include a Content-Signal header indicating how the content can be used:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

This signals that content is available for AI training, search results and AI input (including agentic use). Cloudflare plans to add custom Content Signal policy options in the future.

The Content Signals framework launched during Cloudflare’s Birthday Week last year, letting publishers express preferences for how their content gets used after access.

Tracking usage

Cloudflare Radar now includes content type insights for AI bot and crawler traffic, showing the distribution of content types returned to AI agents grouped by MIME type category.

Publishers can filter requests for markdown by specific agents or crawlers. The data tracks how AI bots, crawlers and agents consume web content over time, accessible via public APIs and Cloudflare’s Data Explorer.

Availability

The feature is in beta at no cost for Pro, Business and Enterprise plans, plus SSL for SaaS customers. To enable it, log into the Cloudflare dashboard, select your account and zone, find Quick Actions and toggle Markdown for Agents.

Cloudflare already enabled the feature on its developer documentation and blog. Other conversion options include Workers AI’s AI.toMarkdown() method (supports multiple document types and summarization) and the Browser Rendering /markdown REST API (for dynamic pages requiring browser rendering before conversion).

Implementation considerations

For newsrooms using Cloudflare, this feature requires no code changes. The conversion happens automatically when AI agents request content with the appropriate headers.

Publishers not using Cloudflare can still convert documents using Cloudflare’s Workers AI or Browser Rendering APIs, though these require integration work.

The shift toward AI-driven discovery means newsrooms should consider how their content appears to AI systems, not just human readers. Markdown for Agents removes a technical barrier to that optimization.

The post Cloudflare now converts web pages to markdown for AI agents appeared first on The Media Copilot.