A year ago, Cloudflare drew a line in the sand against unbridled AI crawling of the internet. Exactly one year later (again on Canada Day) the company took what it says is the next major step on that journey, introducing new tools for publishers and content creators to not just block bots from crawling their content, but charge them for access.
To me, the most interesting part of this is the new Pay Per Use framework. This builds on the existing Pay Per Crawl system, which charged bots whenever they crawled a page. But that straightforward approach didn’t necessarily capture the value of the crawl—once captured by an AI crawler, a piece of content could be used multiple times, in hundreds or even thousands of answers. On the other hand, something could be crawled and never used at all.
Pay Per Use fixes this by compensating the content owner when their content is actually used in an AI answer. Theoretically, if you published something unique, valuable, and optimized for machines to read, it could end up paying dividends for as long as people ask about it. And knowing that is part of the new system, too—Cloudflare promises analytics for content owners so they know how their content is being used. It’s also going to have a better system of telling bots when content hasn’t been updated so they don’t keep re-crawling the same static page over and over.
The system sounds like a sensible evolution to Pay Per Crawl—at least for inference (i.e. AI search engines). For AI training bots, Pay Per Crawl actually strikes me as the better solution since it’s more “one and done.” And how would you measure the value of an individual piece of content in a training set anyway?
All of this depends on a workable payment system, of course, and Cloudflare shared details on how it’s evolving that part of the framework. The new Monetization Gateway is straightforward:
- a bot tries to access content
- the gateway responds with the payment needed and how to pay
- the bot deposits the payment and gets a proof of payment
- The bot then re-requests the content with the proof
- the gateway checks it and bestows access.
It’s all nice in theory, but this kind of usage-based pricing becomes a bookkeeping nightmare on the content owner’s side. This is one of the big reasons micropayments never took off in digital publishing—the revenue from a small payment by a single customer was never worth the processing hassle.
Cloudflare says its unique position as a content delivery network helps solve these problems. It’s already tracking and classifying the bots, so it’s easy to add the payment credential to the process. There’s no “account creation” or anything like that—the bot just shows the receipt. And it’s all done on an open protocol, with no checkout pages or separate payment API. Apparently, there are advantages to managing traffic for 20% of the web.
Cloudflare is refreshingly honest that its new Pay Per Use system is an experiment. How this all plays out depends largely on adoption, not just by publishers but also by AI companies and data brokers. Lots of people often say that digital publishing needs its Napster moment—when the music industry transitioned from sketchy Napster downloads to the “legit” option of iTunes. But iTunes downloads were aimed at individuals. Nobody typing a search into Google or Claude is deciding what content to pay for. This is all determined at the company level, and companies will always choose to get the best/most data for the least cost.
And that will ultimately come down to a simple equation: Is it less costly to get the data they want via Cloudflare’s system? If it’s not, it will remain merely an experiment.
