Cloudflare said Wednesday it will begin blocking AI training and agent crawlers by default on ad-supported websites, a change that could force companies such as Google, Apple and Microsoft to more clearly separate search indexing from AI training if they want continued access to large parts of the web.

The policy, scheduled to take effect Sept. 15, applies to new Cloudflare customers, new sites added by existing customers and existing Free-tier customers who have not changed their settings. Search crawlers will remain allowed by default, but training and agent crawlers will be blocked on pages that display ads.

The company said the changes are designed to help publishers remain visible in AI-powered search results while preventing their content from being used for AI training or autonomous agents without permission or compensation. 

“Now that the majority of traffic is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Matthew Prince, Cloudflare’s co-founder and CEO. 

Splitting up mixed use crawlers

The Web giant said bots that combine search, AI training and agent activity—known as mixed use crawlers—without letting site owners choose among those uses will be blocked on ad-supported pages when training or agent access is blocked. In a company blog post, Cloudflare named Googlebot, Applebot and BingBot as multi-purpose crawlers that could be affected by the most restrictive applicable rules.

“We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training,” Prince said. 

Cloudflare said customers will be able to manage three categories of AI traffic: Search, which indexes content for later retrieval; Agent, which accesses a site on behalf of a user in real time; and Training, which collects content to train or fine-tune models. The controls are available to all Cloudflare customers, including those on the Free tier.

That distinction matters for smaller sites. A spokesperson for  Cloudflare said the new controls are intended to give all website owners more options for managing AI traffic, not only publishers with ads or subscriptions. But the default blocking policy is tied to pages with advertising, and Cloudflare’s compensation plans remain focused on commercial use cases where AI systems access or surface publisher content.

Alongside the new crawler controls, Cloudflare is expanding analytics to show publishers how bots interact with their content and how much traffic AI platforms send back. The company is also pushing into what it calls Answer Engine Optimization, or AEO, offering tools it says will help customers understand how often their content is cited or surfaced in AI-generated answers.

Cloudflare also announced efforts to reduce unnecessary AI crawling. According to the company, more than half of AI crawler traffic is spent repeatedly checking web pages that have not changed. Because Cloudflare sits between websites and online traffic, it says it can signal to AI companies when pages have been updated and worth revisiting. The company said it is testing those signals with AI firms and plans a broader rollout later this year. 

New compensation model

The company is also expanding its publisher compensation strategy by evolving its Pay Per Crawl program into a new system called Pay Per Use. Rather than paying publishers when content is crawled, the new model is designed to compensate them when their content is actually used in AI products. Cloudflare said it is working with AI companies including Ceramic.ai and You.com on the initiative. Under the arrangements, publishers could be paid when their content appears in AI search results or when AI agents access premium content on demand. 

But the model does not yet answer the hardest compensation question: what happens when a publisher’s work is used for model training but never appears in a cited answer? Asked whether Pay Per Use compensates publishers in that scenario, The spokesperson said the program is aimed at “programmatic, real-time access and discovery,” and described Pay Per Crawl and Pay Per Use as only two possible economic frameworks.

“The digital landscape is evolving rapidly,” said Marrissa Holloway for Cloudflare. “We welcome ideas from publishers, creators, and AI companies alike on how to build a thriving agentic Internet.”

Holloway did not directly say what Cloudflare’s cut of any revenue generated would be. “It has always been our philosophy that our customers derive many multiples of value more than they pay us,” she said.

The Media Copilot’s take

Cloudflare is not solving AI compensation for the whole Web. It’s building a bargaining layer for larger publishers with enough traffic and revenue to measure, block and negotiate. That helps the larger content outlets, but smaller sites and independent publishers will get switches to turn on and off. That’s useful, but switches don’t mean they have leverage. The long tail of the Web—the indy blog sites, community web pages and hobby sites—can say “no” more clearly, but there still no obvious way for them to get paid when their work is used for an AI’s training data and never comes back with a citation or link.

The post Cloudflare will block AI training crawlers by default on ad-supported sites appeared first on The Media Copilot.

Anthropic restored global access to Claude Fable 5 on Wednesday, one day after launching Claude Sonnet 5, closing out an 18-day export-control suspension that had cut off both of the company’s newest flagship models since mid-June. Anthropic also restored access to the more powerful Mythos 5 model for a set of U.S. organizations approved under its Glasswing cybersecurity program.

The back-to-back announcements come at a sensitive moment for the company. Increasingly more capable models, like OpenAI’s GPT-5.6 family, draw heightened scrutiny from the U.S. government over national security and cybersecurity concerns. 

The suspension began June 12, when the U.S. government applied export controls to Fable 5 and Mythos 5 after Amazon researchers reported a technique that allowed Fable 5 to identify software vulnerabilities, and, in one case, produce code demonstrating how to exploit one. 

Anthropic said it had found that several less-capable models, including Opus 4.8 and OpenAI’s GPT-5.5, could replicate the same behavior, and that the bypass didn’t expose any capability unique to Mythos 5. The company nonetheless spent the following weeks working with the government to fix it, training a new safety classifier that it says blocks the specific technique “in over 99% of cases.”

Anthropic acknowledged the new classifier could flag more benign coding and debugging requests as it errs toward caution. Newsrooms and media teams running Claude Code or agentic research workflows should expect occasional false-positive refusals on security-adjacent tasks going forward.

Fable 5 is rolling out today across Claude Platform, Claude.ai, Claude Code and Claude Cowork. Pro, Max, Team and select Enterprise plans can access it for up to half of their weekly usage limits through July 7. Then, access will shift to usage credits. 

Sonnet 5, which Anthropic launched late on June 30, is Anthropic’s middle ground between capability and safety. According to the company, the model performs close to the former top model, Opus 4.8, but at a cheaper price: $2 per million input tokens and $10 per million output tokens. The price will rise to $3 and $15, respectively, on Sept. 1.

Security concerns

But the same capabilities that make AI systems more useful can also make them more dangerous, raising questions about whether safety measures are keeping pace with rapidly advancing model performance.  

In its announcement, Anthropic sought to reassure users that Fable 5 has adequate security measures and Sonnet 5 poses a relatively low risk. The company said the Sonnet model is better than its predecessor at refusing malicious requests and resisting prompt injection attacks, a common technique used to manipulate AI systems into bypassing their safeguards. 

However, Anthropic says that while Sonnet 5 exhibited fewer undesirable behaviors overall than Sonnet 4.6, it also showed higher rates of misaligned behavior than both Opus 4.8 and Claude Mythos Preview, which have stricter safety controls. 

“Sonnet 5 was never able to develop a full working exploit, but it does show a slightly higher rate of partial success than Sonnet 4.6. This latter change is likely due to improvements in general intelligence rather than specific training,” Anthropic said in its press release. 

These results suggest that improvements in general reasoning and problem-solving abilities may also increase the model’s capacity to assist with offensive cyber activities, although the company emphasized that Sonnet 5 was unable to develop a complete exploit for Firefox vulnerabilities.

The findings reflect a broader concern among governments and security researchers: AI models do not necessarily need specialized cybertraining to become more useful to attackers. As reasoning and problem-solving abilities improve, models may naturally become more effective at identifying vulnerabilities, generating attack strategies and assisting with technical exploitation.

“Because we judged that the overall level of cybersecurity risk from Sonnet 5 was low, the safeguards are less strict than those launched with Fable 5, which block a much wider range of cybersecurity tasks,” Anthropic said. 

The company said Mythos 5, which is still restricted to a small set of companies and organizations in Glasswing, “can be used to find and exploit software vulnerabilities more effectively than any other model — and all but the most skilled human security experts.” 

Fable 5, however, launched with what Anthropic called the strongest safeguards it has ever applied to a model, after doubling its safety research staff in the month before launch. Researchers from the Commerce Department’s Center for AI Standards and Innovation tested both the original and updated safeguards and, Anthropic said, “agree that they are extraordinarily strong.”

U.S. government collaboration

With the lifting of the Mythos and Fable restrictions, Anthropic is further deepening its cooperation with the U.S. government. This marks an abrupt turnaround for the company, which has had a turbulent relationship with the Trump administration since the Pentagon labeled the company a supply chain risk in late February. The feud arose over Anthropic’s opposition to the use of its Claude models for mass domestic surveillance or fully autonomous weapons systems. 

Reuters reported that Commerce Secretary Howard Lutnick said in a letter sent to Anthropic that the company would work with the government on safety protocols for Mythos, Fable and future models, and to disclose any malicious activity it detects. However, Lutnick warned that the department “reserves the right to reevaluate the decisions made in this letter and the necessity of reimposing a license requirement, should circumstances change or should Anthropic fail to adhere to its commitments.”

“Our hope is that this collaboration … will serve as the basis for systematic rules for the whole industry,” Anthropic said in its Fable 5 release, “and even offer the beginnings of a template for effective global coordination on the risks and benefits of AI.”

The company is currently appealing the supply-chain risk designation in the D.C. Circuit Court of Appeals. It’s unclear how today’s announcement will affect the suit.

The post Anthropic restores Fable 5 access, launches Sonnet 5, as Washington scrutiny deepens appeared first on The Media Copilot.