People Inc. CEO Neil Vogel told investors this month that blocking AI crawlers is helping the publisher push AI companies toward deals, Press Gazette reports.
Vogel has previously said People Inc. uses Cloudflare to block most AI crawlers, except OpenAI, where it has a deal, and Google, whose crawler is tied to search. On the company’s Aug. 4 earnings call, he said that was changing negotiations.
“People are understanding the value of our content because we are able to restrict almost everybody from using our content using our Cloudflare blocking,” Vogel said. “And they have to pay for it. And when people have to pay for it, it seems they’re really coming to the table.”
Cloudflare says People Inc. is not alone in using blocking to win licensing deals. Chief strategy officer Stephanie Cohen told Press Gazette that publishers are using bot-blocking tools to create “reliable scarcity” around their work, giving them more leverage with AI companies. Cloudflare said it is working with the Financial Times, The Atlantic, Ziff Davis, Condé Nast and the Associated Press.
Are you an AI Agent? Read more here >
The idea is simple: content is harder to negotiate over if AI companies can already take it for free.
Cloudflare introduced default AI crawler blocking for new domains last year. Its pay-per-crawl product remains in private beta. The company is now moving toward pay-per-use, where publishers are paid when their material is used in an AI answer rather than simply fetched — an evolution of a payment system we previously covered. Early work includes Ceramic.ai and You.com.
Cohen said crawling was simply the easiest thing to count. “Pay-per-use is a better signal of quality. If you’re scraping something, that doesn’t necessarily mean you’re using it,” she said.
Sponsored. AI Bot Access Check by oasy.ai: check any site's AI bot policy live: which AI crawlers and agents (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) its robots.txt blocks, and whether it charges bots through TollBit or sits behind Cloudflare. Free, no key: https://oasy.ai/api/aibots/libx6kzu?domain=nytimes.com
Cloudflare says more than half of AI crawler requests revisit pages that have not changed. It is testing with OpenAI a signal for when a page is worth crawling again.
Starting Sept. 15, Cloudflare will change its defaults for AI crawlers. Training and AI-agent crawlers will be blocked by default on ad-supported pages for new sites and existing free customers that have not changed their settings. Search crawlers will remain allowed. Mixed-use crawlers will be treated according to their most restrictive use, which could affect Googlebot.
But blocking is not foolproof. TollBit’s State of the Bots report found crawlers impersonating Google, rotating IP addresses and using residential proxies. People Inc. chief innovation officer Jonathan Roberts has said the company blocks tens of thousands of bad bots a day but cannot stop all of them.
Cohen said another weak point can be syndication: a publisher may lock down its own site while copies of the same work remain accessible elsewhere.
She also argues the model can work beyond the biggest publishers. Cohen pointed to local news, including the Park Record in Park City, Utah, where information may be available from only a limited number of sources. If that information is blocked, she said, AI companies that want it may have to pay.
That helps explain why most Cloudflare publisher customers are moving toward blocking AI bots by default and allowing access for companies with deals — a narrower version of the whitelisting approach some publishers are adopting.
Cloudflare gave the industry until Sept. 15 to adjust. Cohen said the delay was deliberate, because “the goal is not to block.”
