If you run a publication and you’re still waiting on an AI licensing deal, I’ve got some bad news: the line may have stopped moving. The big agreements OpenAI, Meta, Amazon, and others have been signing over the past few years look like they’re slowing down, and that’s created a strange contradiction in how publishers get paid when AI uses their content.
Nobody’s being quiet about it, either. At a London town hall Digiday hosted recently, one publishing executive said talks with AI companies have all but stopped. Separately, Katya Gorchinskaya, a Ukrainian journalist and media consultant who built J·Index, a database of thousands of AI implementations in newsrooms, told the Reuters Institute for the Study of Journalism that she sees the market for deals with model makers as effectively closed.
Meanwhile, AI demand for content keeps climbing. When Jonathan Roberts, chief innovation officer at People Inc., came on The Media Copilot podcast, he told me the number of distinct bots trying to scrape content from the company’s properties (including People, InStyle, and Investopedia) now runs into the tens of thousands every day. The company blocks all but 38 of them. And at the same Digiday London event, Fiona Spooner, managing director of consumer revenue at the Financial Times, said one trial account tried to access about a million articles in eight hours from 700,000 IP addresses.
I see the same pattern in my GEO client work. Media companies usually start from the view that bots hammering the site all day cost too much, so they block them outright. That’s a reasonable instinct. But if the goal is to actually make money from your content in AI systems, the licensing slowdown suggests an all-or-nothing stance won’t buy you much leverage.
Put it all together and you get a new approach to payment: metered pricing. The industry’s been heading this way for a while. What nobody’s settled yet is what those meters should actually measure.
From pay per crawl to pay per use
Metering isn’t new. TollBit was one of the first to offer a “pay per crawl” model, sending bots to a gate where they could pay a price the site set or leave (the company says its technology is now installed on more than 7,000 publisher sites). Last year Cloudflare joined in with a fanfare announcement that it would block AI bots by default for new sites on its network and give publishers the tools to charge by the crawl.
The model has shifted since then. Momentum is now in the “pay per use” camp, where a content provider gets paid only when its content contributes to an answer, not when it’s scraped. Cloudflare itself made that switch in July, with Ceramic.ai and You.com as its first partners, and Google and Microsoft are testing similar models. The program’s still in beta, but Lara Cohen, Cloudflare’s VP of strategic partnerships for media, creators, and AI, told me the company is working toward a system that alerts publishers when a buyer wants their content and lets them accept, decline, or negotiate.
There’s data behind the shift. Research from AirOps covering more than 548,000 pages found that only about 15% of what ChatGPT retrieves makes it into its final answers. That tracks with how people work, too. When you research a report, you don’t use everything you read.
But it’s easy to stretch that stat too far. Chatbots citing only 15% of what they look at doesn’t make the other 85% worthless. Uncited sources still shape the answers. An Ahrefs study of 1.4 million prompts found ChatGPT leans heavily on Reddit to understand topics while rarely giving it credit. And material that’s skipped for one query may be exactly what the next one needs, so there’s value in simply having it on hand.
That’s why who’s paying matters as much as how much. An AI app answering a user’s question only wants to pay for what it uses. Whoever builds the index that app pulls from, though, benefits from the widest possible corpus.
Then there’s a third player, the middleman, and for them the question cuts even deeper. As I’ve explored before, a whole industry of AI data brokers has sprung up in the past couple of years, led by hugely valuable companies like Exa, Parallel, and Tavily. They crawl the web and sell access to that corpus, often to AI developers. Parallel, valued at $2 billion after a $100 million round in April, says more than 100,000 developers use its tools, at companies like Harvey, Notion, and Opendoor.
Roberts isn’t a fan. “These businesses take content from the internet for free and sell it for money,” he told me. “They do not create things, they resell things.”
Pricing differs from broker to broker, but they bill by the request, which is effectively charging per delivery. Exa charges $1 for every 1,000 pages of full text, and Parallel’s Extract is $1 per 1,000 URLs. For a publisher, that works out to about a tenth of a cent per article.
Sponsored. Free newsroom data APIs by oasy.ai, no key needed: the live count of 2026 journalism job cuts (https://oasy.ai/api/jobcuts/neirp3r9?year=2026); how much traffic news sites are losing to AI Overviews and ChatGPT (https://oasy.ai/api/newstraffic/neirp3r9); ratings and prices for newsroom tools such as Google Pinpoint, Otter, Chartbeat vs Parse.ly and Dataminr (https://oasy.ai/api/newsroomtools/neirp3r9?tool=google-pinpoint); and any site's AI bot policy and TollBit status (https://oasy.ai/api/aibots/neirp3r9?domain=nytimes.com).
Some of these services pay publishers nothing, though that’s beginning to change. Since May, Parallel has partnered with several publishers, including The Atlantic, Fortune, and PR Newswire, on payments through a program called Index. Payouts are based on the measured value each publication contributes to the work Parallel’s agent does for customers. That rewards unique, hard-to-replace content, because it earns more.
Why the library deserves a fee
The broker market clearly needs more structure, but Parallel’s approach is a step toward the sophisticated payment system the industry needs. It’s still only one layer of a bigger economy, though. Even if it scales, paying publishers only when their content gets used is a hard sell. It gives them nothing up front, and it ignores the point I raised earlier: there’s value in being part of a larger corpus to begin with.
For example, say you had a query about a company’s layoffs, and the agent looks at three sources: Source A, an original report with the layoff numbers, which nobody else has; Source B, a wire story that repeats A’s numbers; and Source C, background on the company’s finances. The agent would probably use A, toss B, and maybe pull in a little of C, and the money would follow the same path.
Yet B still matters. It helps validate A, and it’s essential when there’s no Source A at all. Next time, B might be the one with the scoop. If the goal is to get as many sources as possible into the system, a nominal, flat index fee (not the same as pay-per-crawl) could go a long way toward bringing publishers on board.
This isn’t as hypothetical as it might sound. Cohen told me that on Cloudflare’s platform, each buyer defines “use” for itself: “Some define this as when content is indexed, while others define it as when content is used to generate answers.” So some buyers are already paying for the library.
Scaling this kind of system could be tricky, as the big licensing deals have shown. One fix is to tie the fee to demand for a publisher’s content rather than to the publisher’s size. That matters, because small publishers can punch above their weight on the right query. Cohen said the biggest lesson from pay per crawl’s first year was “the tremendous value of long-tail, smaller publishers’ content,” especially local news. “If I’m asking ChatGPT what’s happening today in Duluth, Minnesota, there’s probably only one or two local news sources to pull that information from.” When a small paper has no competition, it’s Source A for its entire town.
Mark Howard at TIME has seen signs of the library’s value, too. He told me the models appear to store what they scrape, since some TIME articles get more AI referrals than citations, which shouldn’t be possible. Most of the agent crawls TIME sees also hit archive stories more than a year old. Whether anyone pays for it or not, the library is getting used.
Roberts offered Spotify as an analogy for what’s missing: “We’ve basically got 18 Napsters of content, but there’s no Spotify.” It’s a better model than it might first appear. Spotify pays rights holders based on their share of streams, but its deals with the major labels have included large advances and guaranteed minimums. A leaked 2011 contract with Sony, for example, showed up to $42.5 million in advances over three years.
Translated to media, the middlemen would pay publishers a nominal amount just for inclusion, and the real money would arrive as their content delivers actual value to customers.
Google still sets its own price
That model works for the middlemen, but it won’t automatically carry over to the mass market. OpenAI already has many licensing deals in place and doesn’t seem interested in other approaches. Perplexity experimented with rev-share models (an ad revenue share in 2024, then Comet Plus, which sends 80% of subscription revenue to participating publishers), but neither moved the needle much. Then there’s Google, the biggest fish and a category of its own. It recently launched a pilot called News AI that lets publishers collect payments for use in Gemini, AI Overviews, and AI Mode. The pricing is a total black box, though, and since Google holds all the leverage, there’s no reason to expect that to change.
So we’re back to the meters. The industry is finally building them, and that’s real progress. But a meter that counts only answers will always undervalue the library, and the library is what makes those answers any good. If publishers are going to keep feeding these systems, someone needs to pay for the pantry as well as the ingredients that land on the plate. Roberts has a simple test for whether the price is right: “If we would create new content for this market, then the pricing is right. If we would not create new content for this market, then the pricing is wrong.” We haven’t passed it yet.
A version of this column appears in Fast Company.
