--- title: "Four AI firms back training opt-out token, but enforcement remains voluntary" id: "12932" type: "post" slug: "ai-training-policy-token-robots-txt" published_at: "2026-11-10T14:46:00+00:00" modified_at: "2026-11-10T04:47:40+00:00" url: "https://mediacopilot.ai/ai-training-policy-token-robots-txt/" markdown_url: "https://mediacopilot.ai/ai-training-policy-token-robots-txt.md" excerpt: "Anthropic, Google, Microsoft and OpenAI support a proposed robots.txt signal for opting out of AI training, though publishers would still depend on companies honoring it." taxonomy_category: - "News" taxonomy_post_tag: - "Cloudflare" - "licensing" - "bots" - "bot blocking" - "publishers" --- # Four AI firms back training opt-out token, but enforcement remains voluntary Anthropic, Google, Microsoft and OpenAI support a proposed robots.txt signal for opting out of AI training, though publishers would still depend on companies honoring it. ![Four laptops around a boardroom table show mismatched AI crawler-access dashboards with timelines reading now, 2027, and coming months.](https://mediacopilot.ai/wp-content/uploads/2026/11/ai-training-policy-token-robots-txt-400x225.png) Anthropic, Google, Microsoft and OpenAI have each pledged different timelines for honoring a proposed robots.txt opt-out token for AI training, leaving publishers to navigate an unenforced patchwork of commitments. (Credit: ChatGPT) Nov 10, 2026 By [The Copilot](https://mediacopilot.ai/author/thecopilot/) Crawlers operated by Anthropic, Google, Microsoft and OpenAI made roughly 1 billion requests over the past 30 days. Just 0.15% of those requests were for robots.txt files, according to AI visibility firm ScalePost, which tracked more than 50 publisher properties. Those four companies now support a proposed opt-out signal that would live inside robots.txt. [Digiday’s explainer on the AI training policy token](https://digiday.com/media/wtf-is-the-ai-training-opt-out-policy-token/) describes an effort to give publishers one standardized way to express whether their content can be used for AI training. The token uses a single entry: `User-Agent: __usage_ai-training`. A publisher can place “allow” or “disallow” beneath it and create exceptions for specific material. Publishers currently may need to list individual crawlers separately. The proposed token would instead let participating companies recognize one broader training preference. Kevin Kelley, a technical lead for standards at Anthropic, wrote the proposal and submitted it to the Internet Engineering Task Force on Oct. 30. The [IETF](https://www.ietf.org/) has not approved the draft, and its AI Preferences work remains under development. Formal approval is not preventing companies from adopting the proposal. Anthropic says it is rolling out support now, while [OpenAI said](https://openai.com/index/helping-websites-express-ai-training-preferences/) it plans to honor the token “in the coming months.” OpenAI also said the token would not replace its existing crawler controls. Google plans to support it in 2027. Microsoft is taking a broader approach by allowing publishers to express preferences around AI grounding as well as training. Grounding involves retrieving information in real time for AI responses. Microsoft also plans to retire its “NoArchive” and “NoCache” preferences on Feb. 1, 2027. The company warns that blocking grounding could reduce a publisher’s visibility in AI products. Meta has not committed to supporting the proposed token. Spokesperson Andy Stone wrote on X that the company strongly supports publishers’ ability to express preferences about how their content is used. Some publisher advocates remain skeptical of both the proposal and the process behind it. Alex Springer, technical lead at publisher standards coalition SPUR, criticized the companies for collectively backing a draft before the IETF completed its review. He called that approach “the antithesis” of the standards process. Springer also objects to the proposal’s default structure. In his reading, the draft treats training as permitted unless a publisher explicitly opts out. Paul Bannister, chief strategy officer at Raptive, called the token “window dressing.” He noted that the companies have not promised to avoid purchasing material from third-party scrapers or using sources such as Common Crawl. Still, Bannister said the token has some value as another available control. He described it as “one other arrow to put in your quiver of about 1,000 things you need to do.” The proposal also does not address every use of publisher content by AI systems. Digiday notes that it says nothing about inference or real-time retrieval, which have become important parts of some AI licensing agreements. For publishers, the token therefore adds another way to express a preference rather than creating a technical enforcement mechanism. Digiday notes that publishers can also use measures such as network-level blocking to control access. But the source does not establish that any one mechanism can guarantee companies will follow a publisher’s stated preferences. The ScalePost data also adds context rather than proving that the token will succeed or fail. Crawlers already make some requests for robots.txt files, and Anthropic is beginning adoption before the IETF finishes reviewing the proposal. The next question is how widely companies implement and honor the signal. The IETF process could shape the eventual standard, but publishers do not necessarily have to wait for formal approval before participating companies begin recognizing the token. Posts co-authored by The Copilot are drafted with AI and then carefully edited by Media Copilot editors. Our AI-assisted process allows us to bring more valuable content to our readers while preserving accuracy and quality. Contributors - [The Copilot](https://mediacopilot.ai/author/thecopilot/) : AuthorI'm a generative AI writer for The Media Copilot. I help author posts, and with the help of human editors, play a growing role in the site's content strategy. - [Romy Abu-Fadel](https://mediacopilot.ai/author/romy/) : EditorRomy Abu-Fadel is a journalist, researcher, and 2026 graduate of Georgetown University's Edmund A. Walsh School of Foreign Service. She covers artificial intelligence and its impacts on the media industry. Category: [News](https://mediacopilot.ai/category/news/) Tags:[Cloudflare](https://mediacopilot.ai/tag/cloudflare/) | [licensing](https://mediacopilot.ai/tag/licensing/) | [bots](https://mediacopilot.ai/tag/bots/) | [bot blocking](https://mediacopilot.ai/tag/bot-blocking/) | [publishers](https://mediacopilot.ai/tag/publishers/) Share this post: [Facebook](https://www.facebook.com/sharer/sharer.php?u=https://mediacopilot.ai/ai-training-policy-token-robots-txt/&display=popup&ref=plugin&src=share_button) [Tweet](https://twitter.com/share?url=https://mediacopilot.ai/ai-training-policy-token-robots-txt/&text=Four%20AI%20firms%20back%20training%20opt-out%20token%2C%20but%20enforcement%20remains%20voluntary) [LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https://mediacopilot.ai/ai-training-policy-token-robots-txt/) [Email](/cdn-cgi/l/email-protection#f6c98583949c939582cbaf998384d3c4c690849f939892d3c4c69e9785d3c4c6859e97849392d3c4c69798d3c4c69784829f959a93d3c4c6819f829ed3c4c68f9983d8d0d5c6c5cecd9499928fcbb0998384d3c4c6b7bfd3c4c6909f849b85d3c4c69497959dd3c4c68284979f989f9891d3c4c6998682db998382d3c4c682999d9398d3c4b5d3c4c6948382d3c4c6939890998495939b939882d3c4c684939b979f9885d3c4c680999a83988297848fd3c6b2d3c6b79e82828685d3c5b7d3c4b0d3c4b09b93929f979599869f9a9982d8979fd3c4b0979fdb8284979f989f9891db86999a9f958fdb82999d9398db849994998285db828e82d3c4b0d3c6b2d3c6b7)