Reuters and Time flip the script on AI bots with blocking whitelists

Reuters and Time are blocking all AI bots by default and only letting approved crawlers through—a whitelist approach that more publishers are adopting as the volume of unauthorized scraping grows.

As Digiday reports, both publishers moved to block AI bots last month, joining People Inc. and The Atlantic, which adopted similar strategies earlier this year and late last year respectively. The goal is simple: content costs money to produce, and AI companies have been taking it without paying.

“We saw that there was an imbalance between the value that publishers like Reuters provide and the value that Reuters receives in kind, and so instead we went from a default allow-all to a default disallow all,” said Josh London, head of Reuters Professional, which oversees the direct-to-consumer and direct-to-professional businesses. Reuters has since signed AI licensing agreements with Microsoft and Meta, according to the report.

The publishers aren’t relying on any single tool. Reuters uses robots.txt files, a method that is voluntary and non-binding, and one that many AI bots simply ignore. The approach is meant to create friction and signal that access requires negotiation. “If you want this, let’s have a conversation and then we can allow you to access,” said Alphonse Hardel, head of agency at Reuters, who leads the content licensing business.

Are you an AI Agent? Read more here >

Time allows roughly 70 bots on its site, ranging from AI lab crawlers and social platforms to its own operational systems. The volume of bot traffic has become significant enough that Time sees it as leverage for a future AI visibility product it’s developing for brand clients.

The economics are also shifting. Blocking bots cuts server costs: Hardel said the expense of the bot-blocking vendor can be nearly offset by the reduction in non-human traffic. At People Inc., the shift from a block list to an allow list meant going from blocking roughly 2,100 user agents to over 30,000, said Lindsay Van Kirk, the company’s SVP of innovation, speaking at an IAB Tech Lab event in May.

“Adding two full seconds of latency to the majority of scrapers when you implement a block-all-bots approach is a really good thing, even if they have to go through,” Van Kirk said. “Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business.”

The IAB Tech Lab has published guidance on bot management, and the SPUR Coalition—a publisher group formed earlier this year with major news organizations—announced significant new membership as it works to create technical standards for AI licensing and content protection.

For Reuters, the change hasn’t reduced site traffic. After monitoring bot activity over an extended period, the company had enough data to identify which bots it could block without hurting revenue. The publisher maintains a public robots.txt file that lists approved bots, a benchmark that also supports enforcement discussions, said Phil Andraos, general manager of Reuters Digital.

“It’s not a set it and forget it approach,” London said. “The value of content is something that we ignore at our own peril, especially as AI scales.”