Four AI firms back training opt-out token, but enforcement remains voluntary

Crawlers operated by Anthropic, Google, Microsoft and OpenAI made roughly 1 billion requests over the past 30 days. Just 0.15% of those requests were for robots.txt files, according to AI visibility firm ScalePost, which tracked more than 50 publisher properties.

Those four companies now support a proposed opt-out signal that would live inside robots.txt. Digiday’s explainer on the AI training policy token describes an effort to give publishers one standardized way to express whether their content can be used for AI training.

The token uses a single entry: User-Agent: __usage_ai-training. A publisher can place “allow” or “disallow” beneath it and create exceptions for specific material.

Publishers currently may need to list individual crawlers separately. The proposed token would instead let participating companies recognize one broader training preference.

Kevin Kelley, a technical lead for standards at Anthropic, wrote the proposal and submitted it to the Internet Engineering Task Force on Oct. 30. The IETF has not approved the draft, and its AI Preferences work remains under development.

Formal approval is not preventing companies from adopting the proposal. Anthropic says it is rolling out support now, while OpenAI said it plans to honor the token “in the coming months.”

OpenAI also said the token would not replace its existing crawler controls. Google plans to support it in 2027.

Microsoft is taking a broader approach by allowing publishers to express preferences around AI grounding as well as training. Grounding involves retrieving information in real time for AI responses.

Microsoft also plans to retire its “NoArchive” and “NoCache” preferences on Feb. 1, 2027. The company warns that blocking grounding could reduce a publisher’s visibility in AI products.

Meta has not committed to supporting the proposed token. Spokesperson Andy Stone wrote on X that the company strongly supports publishers’ ability to express preferences about how their content is used.

Some publisher advocates remain skeptical of both the proposal and the process behind it.

Alex Springer, technical lead at publisher standards coalition SPUR, criticized the companies for collectively backing a draft before the IETF completed its review. He called that approach “the antithesis” of the standards process.

Springer also objects to the proposal’s default structure. In his reading, the draft treats training as permitted unless a publisher explicitly opts out.

Paul Bannister, chief strategy officer at Raptive, called the token “window dressing.” He noted that the companies have not promised to avoid purchasing material from third-party scrapers or using sources such as Common Crawl.

Still, Bannister said the token has some value as another available control. He described it as “one other arrow to put in your quiver of about 1,000 things you need to do.”

The proposal also does not address every use of publisher content by AI systems. Digiday notes that it says nothing about inference or real-time retrieval, which have become important parts of some AI licensing agreements.

For publishers, the token therefore adds another way to express a preference rather than creating a technical enforcement mechanism.

Digiday notes that publishers can also use measures such as network-level blocking to control access. But the source does not establish that any one mechanism can guarantee companies will follow a publisher’s stated preferences.

The ScalePost data also adds context rather than proving that the token will succeed or fail. Crawlers already make some requests for robots.txt files, and Anthropic is beginning adoption before the IETF finishes reviewing the proposal.

The next question is how widely companies implement and honor the signal. The IETF process could shape the eventual standard, but publishers do not necessarily have to wait for formal approval before participating companies begin recognizing the token.