Cloudflare, Inc. and beehiiv, a newsletter-focused publishing platform, announced a strategic partnership on June 23, which they say will give independent creators greater control over how AI tools access, discover, and use their work.

Cloudflare’s advanced Crawl Control technology is integrated into the beehiiv platform, offering tools to monitor AI crawler activity in real time and manage those crawlers’ access to their content. This integration gives creators two options for managing their digital footprint: publishers can either allow all or some AI search engines to freely crawl their content for greater visibility or completely block AI scraping to protect their archives for future monetization and licensing.

Managing AI bots historically required manually updating robots.txt files or configuring firewalls. The Cloudflare–beehiiv partnership removes these steps, allowing publishers of all sizes, from major media organizations to independent creators, to easily set automated preferences through the platform’s standard dashboard settings. 

Key features of the integration include personalized analytics showing exactly which AI crawlers are accessing their content, which are being blocked, and how much referral traffic they generate, as well as one-click controls to allow or block specific AI models and automatic updates that extend those controls to new crawlers as they appear. 

Tyler Denk, co-founder and CEO of beehiiv, said AI bots from OpenAI, Anthropic, Google, and Microsoft had crawled content hosted on the platform 490 million times. 

“Creators and publishers should own and control their work,” Denk wrote, adding that they “may want to be indexed to maximize AI discovery” or “may want to keep [their] valuable content private and not crawled by AI bots.” 

Matthew Prince, co-founder and CEO of Cloudflare, echoed Denk’s statement: “As the internet evolves, [Cloudflare’s] commitment remains the same: ensuring creators have the tools they need to thrive,” Price said. “This partnership with beehiiv is the next logical step in that mission, giving newsletter operators the transparency and control to navigate the AI era on their own terms.”

AI Crawl Control has now launched in beta for all beehiiv users, giving publishers new visibility into  how AI systems access their content and the traffic those crawlers generate.  

The post Creators get new say over AI scraping through Cloudflare–beehiiv partnership  appeared first on The Media Copilot.

For the first time in the internet’s history, bots account for more web traffic than humans.

Cloudflare CEO Matthew Prince announced the milestone this week, according to Tom’s Hardware, noting that automated traffic has now eclipsed human-generated requests online, months ahead of even his own projections.

“Welp, that happened faster than I predicted,” Prince wrote on X. “Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet’s history.”

According to Cloudflare’s Radar data, bots represented roughly 57% of all HTTP requests as of late April 2026, with humans accounting for the remaining 43%. Bot traffic has held between 53% and 60% in the weeks since. Prince said the actual crossover occurred in the last few months, though the data is messy enough that pinning down an exact date is difficult.

The shift underscores how quickly AI agents have transformed web traffic patterns. Before the generative AI era, bot traffic sat at around 20% of all web activity, with Google’s web crawler serving as the largest single source. Now, AI agents performing tasks on behalf of users are generating requests at a scale that dwarfs human browsing behavior.

Prince illustrated the contrast at SXSW earlier this year: “If a human were doing a task—let’s say you were shopping for a digital camera—you might go to five websites. Your agent or the bot that’s doing that will often go to 1,000 times the number of sites that an actual human would visit. So it might go to 5,000 sites. And that’s real traffic, and that’s real load, which everyone is having to deal with and take into account.”

The reaction to Prince’s announcement was swift. Tech billionaire Elon Musk replied with a single “Wow” to the post.

The full picture is more nuanced. While bots now dominate HTML request traffic—reading pages, scraping content, indexing sites—humans still account for roughly 65% of total web activity when the metric expands to include app usage, video streaming, maps, and social media scrolling. Bots have overtaken humans in the specific act of navigating and reading the web, but not in the broader measure of people actually using the internet.

Cloudflare, which handles approximately one-fifth of all global web traffic, has been tracking the trend closely. The company’s 2026 Threat Intelligence Report also found that bots now account for 94% of all login attempts across its network, meaning only 6% of login attempts come from actual humans.

The crossing point Prince initially forecast for 2027 arrived in 2026. What once required a two-year runway happened in a matter of months.

The post Cloudflare CEO: Bots have overtaken human traffic online appeared first on The Media Copilot.

The same big tech companies that are stripping news publishers of site traffic are now dictating the terms of the emerging AI licensing market, and taking a significant cut in the process, according to a new report from the Open Markets Institute.

The report, “Same Gatekeepers, New Tollbooths: Mapping the AI Content Licensing Market,” argues that publishers are trapped in what the authors call a “double bind.” As Big Tech develops commercial AI products that siphon readers away from news sites, those same companies are positioning themselves as the gatekeepers of whatever alternative revenue streams emerge.

“Big Tech is occupying both sides of the value chain simultaneously,” write the authors, Courtney Radsch and Karina Montoya of the institute’s Center for Media & Digital Governance. They warn that “the deal structures, price precedents, intermediary take rates, and governance norms taking shape now will be difficult to revise once they are normalized.”

The report examines a growing ecosystem of AI content licensing marketplaces. Some are independent startups. Others are built by the very companies publishers are trying to negotiate with:

• ScalePost takes roughly 15% of revenue earned by rights holders.
• Cloudflare, which handles about 20% of global web traffic, takes an estimated 30% cut through its pay-per-crawl marketplace.
• ProRata.ai, which operates an answer engine built exclusively on licensed content, splits subscription and advertising revenue 50/50 with publishers. More than 500 publishers had signed up as of last summer.
• TollBit and Sphere.ai allow publishers to retain 100% of their revenue, instead charging AI companies a separate transaction fee.
• Microsoft’s Publisher Content Marketplace, announced in February, follows a pay-per-use model — but it’s not yet clear how much Microsoft will keep.

The report points to Spotify as a benchmark: despite a 30% take rate, the streaming model allowed music rights holders to earn significant revenue and stabilize the industry during a turbulent transition. The authors argue similar scrutiny is needed for AI licensing marketplaces, particularly when Big Tech is building the scaffolding.

“Regulatory attention is warranted on these platform operators in order to mitigate their data access advantages and ability to set de facto and potentially coercive standards for an industry in which no independent standards yet exist,” the authors write.

The post New report finds wide disparity in AI tollbooths for publishers appeared first on The Media Copilot.

Cloudflare today launched a feature that automatically converts HTML pages to markdown when AI agents request them, potentially changing how newsrooms publish content for AI-powered search and discovery.

The feature, called Markdown for Agents, uses content negotiation headers to detect when an AI system requests a page. When detected, Cloudflare’s network converts the HTML to markdown on the fly before serving it.

This matters because AI systems increasingly drive traffic to news sites, and those systems prefer markdown over HTML. The conversion happens automatically at the network edge, removing the need for publishers to maintain separate markdown versions of their content.

How it works

AI agents add an Accept: text/markdown header to their requests. Cloudflare detects this, fetches the original HTML from the origin server and converts it to markdown before returning it to the agent.

The company provides a simple curl example:

curl https://developers.cloudflare.com/page \
  -H "Accept: text/markdown"

Cloudflare includes an x-markdown-tokens header in responses that estimates token count, letting developers calculate context window sizes or plan chunking strategies.

Why markdown for AI

Markdown’s explicit structure makes it easier for AI systems to process. HTML pages have grown heavier over the years, making them harder to parse. AI agents typically filter out non-essential elements and scan for relevant content — a process markdown simplifies.

Most AI pipelines already convert HTML to markdown as a standard step, but this wastes computation, adds costs and processing complexity, and may not reflect how content creators intended their material to be used.

Cloudflare’s approach moves that conversion to the network edge, where it can happen efficiently at scale.

Content signals included

Converted responses include a Content-Signal header indicating how the content can be used:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

This signals that content is available for AI training, search results and AI input (including agentic use). Cloudflare plans to add custom Content Signal policy options in the future.

The Content Signals framework launched during Cloudflare’s Birthday Week last year, letting publishers express preferences for how their content gets used after access.

Tracking usage

Cloudflare Radar now includes content type insights for AI bot and crawler traffic, showing the distribution of content types returned to AI agents grouped by MIME type category.

Publishers can filter requests for markdown by specific agents or crawlers. The data tracks how AI bots, crawlers and agents consume web content over time, accessible via public APIs and Cloudflare’s Data Explorer.

Availability

The feature is in beta at no cost for Pro, Business and Enterprise plans, plus SSL for SaaS customers. To enable it, log into the Cloudflare dashboard, select your account and zone, find Quick Actions and toggle Markdown for Agents.

Cloudflare already enabled the feature on its developer documentation and blog. Other conversion options include Workers AI’s AI.toMarkdown() method (supports multiple document types and summarization) and the Browser Rendering /markdown REST API (for dynamic pages requiring browser rendering before conversion).

Implementation considerations

For newsrooms using Cloudflare, this feature requires no code changes. The conversion happens automatically when AI agents request content with the appropriate headers.

Publishers not using Cloudflare can still convert documents using Cloudflare’s Workers AI or Browser Rendering APIs, though these require integration work.

The shift toward AI-driven discovery means newsrooms should consider how their content appears to AI systems, not just human readers. Markdown for Agents removes a technical barrier to that optimization.

The post Cloudflare now converts web pages to markdown for AI agents appeared first on The Media Copilot.