When I wrote last month that publishers were fighting the wrong battle against AI, Jonathan Roberts was one of the first people to write back. His view, in short: the bots have not won the front door yet. As chief innovation officer at People Inc., the publisher behind People, InStyle and Investopedia, he has spent more than a year building that door, so I asked him back on the podcast to make the case.
He made it. Then he kept going, into what happens after the door is shut, and that is the part of the conversation I think publishers of every size should hear.
Listen or watch:
Blocking Is Step One
Let’s start with the part we disagreed about. Roberts says blocking most bots is not only possible but routine now. People Inc. has worked with Cloudflare for a little over a year, and it has moved from blocking the bots it could name to the opposite posture: every bot is blocked by default, 38 are allowed through, and humans are untouched. When the switch flipped, the company found itself turning away tens of thousands of unique bots a day, making tens of millions of attempts between them.
The honor system is over. The robots.txt file, in Roberts’s phrase, is now about as useful as a chocolate teapot: a list of banned patrons taped to the door of a bar whose door is wide open. Blocked bots do not go away, either. They drop their names, then impersonate Google or an advertiser, then pretend to be human, and the aggressive ones rent space on networks of hijacked phones and smart TVs. Google worked with the FBI this summer to shut one of those networks down.
“If you think you’re blocking because you put a block into your robots file, you are not blocking.”
The point for a smaller publisher is that a year ago this was custom work, and now it is off the shelf. Make a list of the partners you want in (ad tech, search, your email and social tools), hand it to your CDN, and block everyone else. The CDN does the rest.
The 18 Napsters
The thing is, blocking is step one. Step two is a price, and that is where the conversation got interesting.
Roberts describes a layer of companies most people have never heard of, names like Parallel, Exa, Tavily and Firecrawl, about 80 of them by his count, that take content from the web for free and sell it to AI developers for money. They are the picks-and-shovels businesses of the AI boom, and they are raising serious venture capital. By his figures, Exa went from a $900 million valuation to $2.2 billion in roughly a year. His summary of the market: 18 Napsters of content and no Spotify. Nothing is paying the creator.
That is also his answer to why there are so few lawsuits in this corner of the business. It is simply too new. The stealth-bot bills moving through Congress and New York State suggest the law is catching up, but People Inc. is not waiting for it.
Instead, the company is trying to make the legitimate route the better product. Its content is available to AI companies through an API as Markdown rather than HTML, stripped of navigation, ad copy and code, which cuts the tokens a model has to reason over by 80% to 90% and comes with cleared rights. Roberts’s framing is the oldest fix in technology: Turn it off, then turn it back on again.
“Do you want to spend your life being Napster? Because there’ll be a lot of Napsters. Or do you want to be a Spotify?”
Who Sets the Price
I asked who gets to set the price, because from where I sit it still looks like a buyer’s market. Roberts’s test is simpler than any rate card: If publishers would create new content for this market, the pricing is right. If they would not, it is wrong.
On the per-crawl versus per-use debate, he thinks it is too early to settle, and he goes back to first principles. If you want the content, you pay for it. Paying comes with specific rights, using those rights means you owe something, and breaking them means you breached the contract. He also notes that the resellers already charge their own customers per delivery, which makes the “we only pay for what we use” argument harder to take seriously.
The Fuel Problem
The claim that will outlast the episode is about inputs. Roberts’s line is that models without reference are fiction writers: very good at sounding plausible, with no requirement to be correct. Give the same model an accurate document to reason over and it becomes a researcher. He goes further, and this is the claim worth arguing about: For most questions, the inputs now matter more than the model.
“You’re spending hundreds of billions, if not trillions, of dollars on making better models, and you’re spending effectively zero dollars on better inputs.”
That is why he wants information treated as a supply chain to invest in rather than an asset to be mined. The training-era models were built on whatever happened to be on the internet. The questions people now ask these systems did not exist a few years ago, and the answers do not exist yet either. Someone has to go find them. There is an industry for that, and it is called journalism. Fund it and the AI gets better. Starve it and the AI gets worse.
Automate the Chore, Not the Choice
We closed on TIME’s experiment with Mobian, which serves ads to the bots crawling its pages (I went deep on that with TIME’s Mark Howard), and Perplexity’s objection to it. Roberts sees the same problem from the brand side: AI is now an intermediary between a company and its customers, and it misrepresents brands often enough that People Inc. runs a program helping advertisers get accurately described by the models.
His bigger worry is what the chatbots are doing to choice. We asked for machines to automate our chores, he says, and they automated our choices too. Search is humble and lets you pick your path. Chatbots are arrogant and pick for you. For toilet paper, fine. For an election or a health question, less so. And, he argues, that is a product decision, not an inevitable feature of AI.
Roberts thinks the next couple of years will decide which of two futures publishers get: a knowledge economy where the people producing information finally get paid for it, or a mediocre web full of AI slop that nobody funded. His first step is the one we started with. Stop them taking it for free. The second is to find the good actors and build the market with them.
The door is closing. What gets built behind it is the real question.
In this episode
Pete and Jonathan discuss:
- Why Roberts says blocking most bots is possible, and now off the shelf
- People Inc.’s default-block posture: 38 bots allowed, everything else out
- The disguises blocked bots wear, from spoofing Google to hijacking smart TVs
- The resellers taking publisher content for free and selling it for money
- 18 Napsters, no Spotify, and why the lawsuits have not arrived yet
- Markdown through an API: making the legitimate route the better product
- Who sets the price, and per crawl versus per use
- Models without reference are fiction writers
- Why inputs matter more than the model, and where the money is going instead
- TIME, Mobian and Perplexity, from the brand side
- Automate the chore, not the choice
- Two futures for publishers
About the guest
Jonathan Roberts is chief innovation officer at People Inc., the publisher of People, InStyle, Investopedia and dozens of other brands, where he leads the company’s AI partnerships, data strategy and bot-blocking work. He last joined the podcast in April 2025, when the company was still called Dotdash Meredith. People Inc.
About the show: To explore more conversations like this and see what’s new, visit the Media Copilot website at mediacopilot.ai. You’ll find new episodes, expanded resources, and tools designed for journalists, communicators, and media leaders navigating the fast-changing world of AI. It’s the home base for everything Media Copilot and it’s just getting started.
Enjoyed this episode? Subscribe to The Media Copilot on Substack, Apple Podcasts, Spotify, or your favorite app. On YouTube? Tap the Like button and Subscribe to the YouTube channel.
Produced by The Media Copilot.
Music: “Favorite” by Alexander Nakarada, licensed under CC BY 4.0
All rights reserved. © AnyWho Media 2026
TRANSCRIPT
[0:02] Pete Pachal
Hi, welcome to the Media Copilot. It’s a podcast about how AI is changing media, news, and communication. My name’s Pete Pachal. I covered tech for a long time as a journalist, and now I have conversations with the media people, the builders, and the creators who are all working to answer the question how will we get information in the future? And what does that mean for the people whose business is information, especially the media? Okay, so we all know what’s happening.
AI systems are ingesting the web at scale to answer people’s questions and now to a certain extent, even helping them do things online. The problem for publishers, of course, is that they’ve spent years, in fact, decades, building businesses around people visiting websites. So another way to look at that is that there’s definitely still demand for information. It’s just moved upstream in towards the platforms. And so the hard part.
For the media is turning that demand into some kind of payment. It sounds straightforward, it’s obviously easier said than done, and it involves navigating the worlds of bots, blocking them or not blocking them, being open to new partnerships and business models, and obviously facing some of the harsh realities about the old system and how it’s not coming back. So, my guest today is exactly the guy you want to talk to for advice on all of that. It’s Jonathan Roberts, Chief Innovation Officer at People Inc.
The publisher behind brands, including People, InStyle, and Investopedia. Jonathan and I have been debating a bit on how much publishers can accomplish by blocking. People puts a lot of serious resources into blocking unauthorized bots, and I want to hear about how that’s going and what small publishers might be able to take from what a larger media company has learned. And there’s already a business of actually selling information.
to AI companies and even developers and other companies. Today I want to talk about what it takes for more of that money to actually reach the people creating the information. Also who gets to set the price of this market, and then what publishers might be able to build for their audience along the way.
[2:15] Pete Pachal
Excited for this one. But a quick note, if you’re listening on Apple or Spotify, it would be great if you could leave a rating or review or both. And if you’re watching on YouTube, please like the video and subscribe to the channel. Those things really do help more people find the show.
[2:30] Pete Pachal
Jonathan Roberts, welcome back to the Media Copilot.
[2:34] Jonathan Roberts
Thank you for having me. It’s great to be here.
[2:36] Pete Pachal
Nice. Yeah, you were here, I think it was about a year and a half ago. And a lot of the stuff that we’re gonna talk about today was very nascent or even maybe non existent. And I know you guys have been working really hard about adapting to the new realities that I was just talking about. Alongside a lot of partners. And so I know the we just have a lot of catching up to do. So I’m really glad you’re here. And so I
I’d love to start with what I was just talking about earlier because I know we were just honestly emailing about this when I published a column about it. So specifically the bot blocking and what I was referring to there, I know you wrote sort of the foreword or the introduction to an industry report about bots. So this was something from TollBit from I think a month or two ago. And it was great because you revealed a lot about how People Inc is dealing with this issue of both you know, good bots and bad bots, I guess we could think about them.
But when I read it and you talked about how sort of successful you’d been, but not entirely successful about it. And that sort of was somewhat reassuring in that it’s possible to block bots, but I also thought, well, People Inc. is like probably the best equipped one of the best equipped places to do blocking, and even they can’t get all of them. So I’m curious. How can you assure me that that this is something where there’s a better takeaway for smaller publishers and folks that may not have the resources of a People Inc.? What can they what should they think about this? Cause my view was kind of like, well, that’s tough and the bot blocking is not that’s not that you would want to let all the bots in, but is that really where the battle should be fought, I guess, so to speak, for all the things I was talking about earlier. I know it’s kind of a big question, why don’t you start with the blocking and then let’s expand out?
[4:29] Jonathan Roberts
Yeah, and I’ll start with a very quick answer, which is it is definitely possible to block most of the bots, right? And so and it’s not that hard and there’s services that will even do it for you for free, right? So like as a starting point, like yes, you can do this, yes, it’s a good idea, yes it has an impact. I’m actually gonna start one step further back, which is kind of to the point in your introduction, is there gonna be valuable information in this kind of new information economy that AI is creating, mediating, moving into?
[4:42] Pete Pachal
Nice.
[4:58] Jonathan Roberts
We see so many people, bots, companies attempting to take our content for free every day. There is a massive rabid appetite for our content. So there is definitely more need for more content in the future than ever before. Right? That is definitely true. The future needs a lot more information, is much hungrier for it than ever before. Now we just need to build a market that makes it worth creating to that need, right? So
In any new market, the question is, is there demand and is there supply? Right? And can you connect them? What I can tell you is like there is definitely demand. Like the it the extent to which people go to try to steal our content is unlike anything I’ve ever seen. So, yes, there is extreme demand for content. Now, if you don’t stop people taking it for free, then there will be a market, but you’re not gonna get paid. Right? So, step one in making a market, stop them taking it for free.
[5:51] Pete Pachal
Right.
[5:55] Jonathan Roberts
Step two, figure out the price, right? but a lot of people are still at step one. Right? And so to talk about exactly how we do that, right? So what is a bot? What’s it doing when it comes to take your content? Basically, a bot is any system that comes to your website that isn’t a human, right? So if somebody writes like a little bit of Python code and like it goes off and pulls sucks down your the HTML of your pages and takes it off to do something, like that is a bot.
If it’s a bot that comes in and uses a browser and like scrolls the page, that is also a bot. What we’re seeing at the moment is the internet used to be running an honesty system, right? Where you would just like put a bot you are no, but like there was the robots that so there’s a file. This got pretty gets pretty nerdy pretty quickly. There’s a file called robots.txt, right? And robots is like a list of the bots.
[6:39] Pete Pachal
Well, some of it.
[6:47] Pete Pachal
Yeah, it’s all right.
[6:53] Jonathan Roberts
That you let in and a list of the bots that you don’t let in. It’s a permission list, right? But it has the effectiveness of like if you were running a bar, sticking up a list of all the people who aren’t allowed to come into your bar on the front door, but leaving the front door open. I mean it’s like, okay, these people who I claim are not trustworthy people, they’re not gonna walk in because I’ve put their name on a list. And anyone who’s name
[7:13] Pete Pachal
And then you also maybe not even training the bartenders to look out for them, you know, like not there’s no second layer here.
[7:18] Jonathan Roberts
Exactly, right? And yeah, and nobody is abiding by that list real really anymore. Right. If you see the files that came out from the New York Times OpenAI case last week, like it is clear that some of the biggest companies in the world decided to totally ignore that and just strip mine the internet, right, for assets, right? So if you think you’re blocking because you put a block into your robots file, you are not blocking.
To use a term from my homeland, robots is now as useful as a chocolate teapot. Like that is not gonna get it done. So you need to now do something with teeth, right? You need to actually start not just saying don’t come in, you actually have start locking the door. Right now, what we can do on our side is limited, but what we can do with our partners is much more successful, right? And so we work with Cloudflare, we’ve worked with Fastly, we’ve worked
[7:53] Pete Pachal
Ha ha.
[8:17] Jonathan Roberts
Akamai, we work with CloudFront, right? And these are companies that, you know, three years ago you would never have heard of. They weren’t they were like how you would stop getting kind of a takedown attack from bots, right? But these are now the front line of protection for IP. What they will do is say, okay, for anybody who comes to your website, is this a person or is a robot? Right? If you’ve ever come across, as we all have those are your robot pages, it’s because of one of those four companies.
[8:46] Pete Pachal
Yeah, I think we’re all seeing those a lot more often.
[8:48] Jonathan Roberts
Well yes. Because it’s a lot less clear whether or not the internet is full of people or the internet is full of robots, so there’s a lot more of them around. What we work on with Cloudflare, who is the partner we’ve been working on this with for a little over a year, is to begin with we just blocked all of the bots that we thought were bad that we wanted to stop. But that requires that they or assumes that they name themselves, which is again back to an honesty system. Tell us who you are and we’ll tell you whether you’re allowed.
That didn’t work because again, it the internet is full of pirates, it’s very dishonest. So now we’re in a situation of assuming that every bot that comes to our site is a bad bot and we’ll block all of them. Right? Apart from yeah, it’s a big paradigm shift. So rather than saying the internet is kind of default accessible for everyone and everything, we’re still completely accessible for all humans.
[9:32] Pete Pachal
Interesting. Okay. So that’s sort of a big paradigm shift from before. It’s not even just if.
[9:48] Jonathan Roberts
That hasn’t changed. But we are fully locked down if you are a bot. Right? There are 38 bots that we allow list, right? And everyone else will block. Short list. I mean, could still be shorter. And what we found when we switched was we started blocking tens of thousands of unique bots per day.
[9:58] Pete Pachal
Well, short list. I mean, considering the number of bots.
Yeah.
[10:17] Jonathan Roberts
Versus the 38 that we allow. Right? And those tens of thousands were attempting to take our content tens of millions of times a day. Right? So there is a huge appetite.
[10:30] Jonathan Roberts
So the fun thing you find out when you start blocking these bots is they don’t give up. They just try harder, right? And so the first thing they do is they come in and they say, Hi, I’m this bot, here’s my name, and you’re like, Great, you’re not on the list, you’re not coming in. Fine. All fine. Then they come and pretend to be anonymous, right? So, you know, they stick on a cap with some sunglasses and try and get through and don’t tell you who they are, and you block them, you’re like, You’re still a bot.
[10:53] Pete Pachal
Ha ha.
[10:56] Jonathan Roberts
Then what they do is they stick on a pair of glasses on a funny nose and they pretend to be somebody completely different, right? So they’ll come in and say, Hi, I’m Google, right? Or hi, I’m an advertiser, right? And so they’re gonna try and pretend to be another bot you that is on the allow list. And so you have to identify them as not being that actual bot and block them, right? Which we do. And we tell the companies that they’re pretending to be that these companies over here are spoofing them, right? Because nobody wants that.
Then they try to be human, like scroll the page, hit a interstitial saying, Are you a robot? They get blocked. And then the really aggressive ones go one step further and they attempt to hijack individual peoples like your mine, you know, your mom’s internet, right? And so they’re coming through people’s personal phones, they’re coming through people’s personal home internet addresses, they’re coming through people’s smart TVs, like Samsung. Yeah.
[11:53] Pete Pachal
Yeah. This is like the botnet thing. I remember reporting on botnets when I was a tech reporter back in when it was you ever the term internet of things was in vogue and yeah, there were there was problems and those are real like hacks. And now this seems like it’s just kind of like a thing people can but bad actors can sort of use when they needed to.
[12:11] Jonathan Roberts
You you can just rent space on a botnet which is made up of people’s smart TVs, right? That that’s wild, right? But that’s a kind of tactic that gets on like national security radars, right? So Google over the summer Google over the summer did something really interesting, which is they shut down one of these they’re called residential proxy networks, right? Which is like lots of people’s home internet being hijacked by other actors, right? So they shut down one of these
[12:14] Pete Pachal
Hmm. Yeah.
[12:25] Pete Pachal
Mm. So it’s a last resort.
[12:40] Jonathan Roberts
Networks called NetNut, right? And they did it in partnership with the FBI. Right? Now, there’s a lot of people say, well, the internet’s free to use, we’re not doing anything wrong. Like if you start using a network which falls in the same category as stuff that the FBI shuts down, I think we’re way past kind of normal behavior or normally acceptable behavior. But when I say these companies have a unlimited appetite for content.
[13:09] Pete Pachal
Hm. Yeah.
[13:09] Jonathan Roberts
Not kidding, right? They’re really aggressive. Now we can block everything kind of up to that last step, and even that last step, like we’re we’re getting pretty good at with Cloudflare, Cloudflare’s an expert in this, the Akamai, Fastly, CloudFront are all working on this problem. Google’s even shutting this stuff down. So what I’d say is even though it is hard today, right, to block even that layer, it’s not impossible. And there’s a bunch of stuff coming which makes that much easier and more straightforward, right? Now
[13:20] Pete Pachal
Mm.
[13:38] Pete Pachal
Can I stop you just there in terms of the blocking, it sounds great spelling out all these things and what they do. So if you are a smaller to mid sized publisher who doesn’t have the same kind of resources, like because it’s you know, some of this is you, some of this is you with partners, how much of this can they replicate in terms of the whitelist, the and the making sure people are human, even stopping the human stuff is
[13:39] Jonathan Roberts
Please.
[14:06] Pete Pachal
Is that stuff that is almost all they can sort of outsource to a cloud flare or like what do they do?
[14:11] Jonathan Roberts
This time last year it would be very custom. This time this year it’s pretty off the shelf. Right. Again, like w we’ve gone so far down the path on this, but nobody’s building it just with us for us, right? Like all of this stuff is now available to the broader group. Really the only thing you need to do is say, who are the people who are coming to my content who I want to let through? Right? Everyone has different sets of partners, right? We have a bunch of advertising partners who need access to those pages. We
[14:15] Pete Pachal
Mm-hmm.
[14:38] Jonathan Roberts
Allows search indexes to come through, but you know, we limit what they’re allowed to do. So we have a list, right? And like our email team needs some, our social team needs some partners, right? And so you build that list by going to the different groups. Everyone has that list or everyone could make that list. And then you go to somebody like a one of the CDNs and say, Here’s the list I wanna let through. Block everyone else, right? And they will help you.
[14:59] Pete Pachal
And is it to some extent now y is sort of the go to thing still the robots.txt? It’s just now the C DN can take that and turn it into something with teeth? Is there more to it than that?
[15:09] Jonathan Roberts
Ha ha
Sorry. There are two different things. Like robots.txt is not necessary to do the blocking. You can just hand your CDN a list, right? Nobody else needs to see it. Right? Or you can say, I’m gonna make my robots file the same as my block list and my allow list, right? Both are fine, but if you’re just relying on robots, it’s not working. If you’re working with the CDN, it can’t.
Ha ha
[15:38] Pete Pachal
So if no, so I think we’ve established the blocking pretty well. I mean, I think we could get a leave even more nerdy and sort of talk about the types of bots, but it but I think that’s going to relate to my next question where I want to do this, which is to say, okay, you there’s a lot of blocking, there’s a lot of unauthorized traffic. And we can get into whether that’s shrinking or growing. I believe it’s growing. I just saw sort of a report. Yeah, from DataDome that just talked about again, they’re they’re about blocking tech, but they had a lot of
[15:58] Jonathan Roberts
Definitely growing.
[16:05] Pete Pachal
Dismaying stats, shall we say, about like the rise of not just bots but bad bots, specifically the kinds that sort of put on the disguises that you talked about. But I’m also thinking about like specifically those sort of 38 that you allow and whatever it’s, you know, that might be different for different publishers, but it is like, so how do you turn that into something that is going to, you know, give you good business outcomes? And what are the various paths towards that behavior, whatever it is,
And turning into something that is helpful. And also, I guess let’s even start like even on the blocking part of it. It’s like because you’ve sort of aggressively blocked this stuff, has that turned into like do you see a path like, this person was in the gray market and didn’t do it, but now we’ve negotiated there allowed and now we’re actually getting seeing good results from that? Like I guess sort of get piece that all together for.
[17:02] Jonathan Roberts
This kind of gets to the question of how do you turn blocking into a business, right? Or how does blocking help you create a business?
Step one is knowing there’s demand, right? And seeing this huge appetite for what we’ve got was good to see. Effectively what we now see is there’s a wealth of companies that have popped up that are not AI companies, but they are getting content for AI uses, right? So they’re kind of the picks and shovels piece of the AI industry. Now if AI needs three things to run.
Right. You need chips, obviously. You need power and you need information. Now the chips are well funded, right? Like you can look at your 401(k) and NVIDIA’s growth has done a lot for that in the last, you know, two, three years. There’s a massive investment in power infrastructure around the entire world to power those chips. But if you’ve just built data centers and you don’t have anything to put through them, you have empty data centers, right? Like
The chips and the power is the engine. The words are the fuel. And in the
[18:13] Pete Pachal
Yeah. It’s a whole garbage in, garbage out idea. Doesn’t matter how good the system is in between, it’s it is what it is.
[18:19] Jonathan Roberts
And in the course of the last eighteen months, a lot of companies have popped up to sell the fuel.
Right. So these are companies that you’ve likely never heard of. They’re companies called like Parallel, Exa, Tavily, Firecrawl, right? There’s a list of about eighty of these companies, and there’s probably a new one every couple of weeks. They are raising huge amounts of venture capital, right, to build these businesses. And these businesses take content from the internet for free and sell it for money. Right? Like that is what they do. They do not create things, they resell things.
and at the moment they’re taking one hundred percent of the economics for themselves. Right? Now, taking things for free and selling it for money is a good business, it turns out. So they’ll keep doing it until they stopped. Right? And so that’s the reason for these blocks. It’s very, very similar to the Napster era, right? We basically got 18 Napsters of content, but there’s no Spotify, right? Nothing is paying the creator.
[19:20] Pete Pachal
So Napster is a like a you know, this analogy, I know you’ve said it before and it’s a great one. But Napster was basically sued out of existence, right? Like and it just couldn’t sustain its model while as any kind of legit business. So so now like I’m curious, like there’s obviously a lot of lawsuits in this field. And you hear about the ones, you know, the New York Times one is the most famous, and but there’s several others.
And but you don’t hear about a lot of lawsuits in the that sort of dark or gray market that you just talked about. Why not?
[20:00] Jonathan Roberts
It’s very new. Right? I mean Exa raised whatever eighty five million at nine hundred million valuation this time last year, and then raised two hundred fifty million at a two point two billion dollar valuation in the spring, right? They were very this whole industry was nascent and small and then it clearly found product market fit. Like you don’t go series B to series C at that kind of fundraise until you’ve really found demand. So this is an industry that has just woken up and come on.
Right. And a bunch of the publishers are identifying it, seeing this, building the blocks, figuring out how we can stop this. I mean, let’s be clear, there’s also bills going through Congress and through New York State, like stealth bots bills, which are talking about
[20:42] Pete Pachal
Right. The legal side, regulatory side, yeah.
[20:47] Jonathan Roberts
Running an anonymous crawler for AI purposes is something that would become illegal, right? If those bills went through. So the fact we’ve already got to the point of like legislation’s like bipartisan legislation getting proposed, this is actually moving quite quickly. And
You know, we talk to these companies. We say, Do you want to spend your life being Napster? Because there’ll be a lot of Napsters, or do you want to be a Spotify, right? And clearly the future is in actually a better permissioned, liability cleared, rights-cleared service, right? Napster died for two reasons. One, because there was always headwind, there were the lawsuits, but also because their service was getting degraded. And then somebody just built something better. Right? And the
You can have the carrots and you can have the sticks, but we need to actually live in an industry where doing this with like liability cleared, rights cleared, quality content that is better is better. Right? And at the moment, you need to get our content, you’re gonna have to run through somebody’s hassle proxy, and it’s gonna take you five, ten seconds, you’ll get blocked eighty percent of the time, and you won’t have you’ll be taking on a liability when you use our content. Or
[21:59] Pete Pachal
Sponsored. Free newsroom data APIs by oasy.ai, no key needed: the live count of 2026 journalism job cuts (https://oasy.ai/api/jobcuts/s7rzbz4k?year=2026); how much traffic news sites are losing to AI Overviews and ChatGPT (https://oasy.ai/api/newstraffic/s7rzbz4k); ratings and prices for newsroom tools such as Google Pinpoint, Otter, Chartbeat vs Parse.ly and Dataminr (https://oasy.ai/api/newsroomtools/s7rzbz4k?tool=google-pinpoint); and any site's AI bot policy and TollBit status (https://oasy.ai/api/aibots/s7rzbz4k?domain=nytimes.com).
So the idea is like making it too expensive. Basically it’s gonna be more cost prohibitive to try to get it the bad way and you’re gonna get poorer quality stuff than just pay us or go on these payment rails or whatever, work something out, right?
[22:14] Jonathan Roberts
There is a better, faster, more efficient way to do this in the way everyone’s doing it. Right. And so, you know, we’re building out custom services to make access to our content fast and efficient and accurate, right? Where if you get the content that way, it’s clean, very, very efficient. You don’t have to go clean it up and we have clear rights around it, right? So if you ha get to do this 95% faster and it’s a hell of a lot cheaper to process over it, that’s a service built for the AI inference feature.
Not trying to hack the old web to try and make a bad service that won’t persist.
[22:49] Pete Pachal
Nice. This is a great element to sort of creating the market and sort of what responsibility might lie in the publisher, I think, which is interesting. So are you speaking, let’s get just to get a little more specific. Are you talking about a machine readable version of your site that is obviously g guarded and walled away? Or are you also talking about like an agentic layer? Is that sort of the same thing? Just give us a little more texture there.
[23:11] Jonathan Roberts
Yeah, so the simple argument here is I think we’ve we’ve all used we’ve all used chatbots now, right? We’ve all used answering, we’ve all used AI. If you put in a really long-winded prompt and you throw a bunch of random junk in there, you’re gonna get a really weird answer back. Right? Now think about a web page. A web page is mostly really random junk, right? It’s a lot of code, it’s a lot of HTML, it’s like JavaScript stuff.
Even the content of the page has like navigation links, ad copy, like there’s all this stuff in there, which is a distraction to a human, we’re good at ignoring it. It’s a distraction to a chatbot, they’re bad at ignoring it. Right? And so if you just like rip a page and drop it in and reason over it, you’re gonna get some weird stuff, right? And remember, these chatbots make things up even when they have good inputs. So you throw some random stuff in there, like it gets pretty bad pretty fast.
And you pay per token you’re reasoning over. So all that extra junk in there is costing the end consumer who’s who’s using it money, right? Token maxing is a bad idea because you burn a lot of money for no purpose, right? Instead, you can get what we’ve created as a service which isn’t HTML, right? It’s Markdown, right, which is the format bots really, really like.
[24:22] Pete Pachal
Mm-hmm.
[24:32] Jonathan Roberts
And it is just the core content of the page, right? The important stuff, who wrote it, when was it written, when was it updated, like no ad copy, no navigational stuff, no JavaScript, no HTML, no nothing, which reduces the amount of words you have to worry about by, you know, 80, 90%, right? And it’s vastly more accurate and faster to reason over, right? So faster, better, more efficient, I mean that’s a better product. Now you can only get that through the API.
[25:01] Pete Pachal
I see. So it’s not something a human would ever see. Yeah. ‘Cause I’ve seen both. People do it through the API, but also some people will put out there on the web a marked down version of their site.
[25:12] Jonathan Roberts
Yeah, if I have a better product, why am I giving that away for free?
[25:15] Pete Pachal
Right. Yeah, makes sense. I mean to be yeah, and to be clear, when I do see that it’s more often associated with a brand than a publisher. So they’re more sort of interested in sort of GEO and sort of being discovered.
[25:28] Jonathan Roberts
Yeah, but I mean this is the like how do you fix things on the internet? All technology is fixed the same way. Step one, turn it off. Step two, turn it back on again, right? So blocking is the turning it off, but actually building a custom service for these use cases is the turning it back on again. Like there’s a better way to do this.
[25:36] Pete Pachal
Ha ha.
[25:45] Pete Pachal
Nice. So let’s let’s then this is a great n transition to the whole question of pricing, right? Which is like how do you have, you know, like you say, you have a market, you have a buyer or seller, you have a means to get the thing to the from what the buyer’s selling to the seller. That sound you know, what that’s all what you were just describing. So then it’s like, well, who’s determining the price? ’cause to some extent, like you know, we’re talking about platforms like intermediaries here, whether it’s, you know, quote unquote legit bots or
something else. And so right now it seems to me it’s it’s mostly a buyer’s market. And I think that’s that depends on I think this was probably a good point also to think about like the different types of content, like you know, sort of more evergreen type stuff, like I don’t know, recipes or whatever, versus real time information, what’s more valuable to which platform and vendor, etc.
So that gets complicated, I know, but like this does get into the pricing, and this is where in a real way, I you know, you hear about these things are sort of nascent, which is true, but it also means like the pricing is kind of low. So w is that good is that a good sign that there’s a price there’s money to be had at all and then maybe it’ll get bigger, or like what do we how do we how does the actual sort of amounts get to a point where this is a real business?
[27:06] Jonathan Roberts
So let’s go back to Exa as an example, right? They raised a $2.2 billion valuation. They do not believe content is free. Right? So there is a real market and there’s a huge amount of money being thrown at it, right? So the question then isn’t a matter of whether there’s a market that people will pay for, it’s who’s getting what share of the money, right? So this market has on some level already been priced to begin with.
Right, ’cause there are all these services that exist who are selling the content for money, right? And those prices are just rate cards. So people on the buy side are already used to paying for content, but they’re paying those prices when the cost of goods for those companies is zero, right? Or minimal app to pay to go scrape it, right? So
[27:57] Jonathan Roberts
There is still a early days on figuring out what the market will bear and what the overall like market end-to-end is gonna look like. There’s lots of different models, right? And this is not a settled question. Like some people have talked about Rev share models, some people have talked about like per use or even per crawl models, right? Some of those come with like minimum guarantees, some of these are more the
the deals that get the headlines, right? The big partnerships between the big publishers and the and the big AI companies that are these kind of multi year, all you can eat. Some of those deals include training, right? So there’s lots of different questions of what the right market structure’s gonna look like.
[28:40] Pete Pachal
Can we double click on the per use versus per crawl for a second? Because this comes up a lot and it’s just kind of usually noted as like there’s two different models. But I’d love to sort of like get your thoughts on what a theoretical. Maybe you’ve been in some negotiations like this where it’s like a scraper, well, some whatever one maybe one of these companies is like, okay, fine, like we’ll figure out something here, but you we can’t do it per crawl because when we our queries with 85% of it’s thrown out. I think that’s a stat I read in some.
Chat GPT stat or whatever. Like most of the stuff it scrapes actually never makes it into the answer. So I’m you know, devil’s advocate, like is there a reasonable argument? Like, look, why don’t we just do it per use? And because it seems to me like a bot blocking sort of stance almost by inherently implies a paper crawl, you know? And maybe the agreement has both layers to it. I don’t know, but
This is I imagine you’ve been in conversations that sort of have to do with this, like what is the right sort of answer in that kind of negotiation?
[29:43] Jonathan Roberts
I think at this point we’re probably in a place of it’s too early to figure this out because the tech is still evolving, and you just go back to first principles and say, if you want our content, you’re going to pay for it. And if you’ve paid for the content, that comes with p specific rights. And if you use those rights, you owe us. And if you break those rights, then you breach the contract. Right? Again, like
B when you get into a world where this is permissioned and contracted, you can have all those conversations. But at the moment, we’re in a world where people aren’t paying for it because they’re just taking it for free. Right? So you we need to get the industry to the other side of that, but they are charging their customers on a per use basis. Right? You can go to these guys and say, Great, I’m gonna pay
[30:22] Pete Pachal
Right.
[30:33] Jonathan Roberts
X dollar per thousand searches. Here’s my credit card, and you’re gonna give me full content of Publisher X’s content when I pay you. Right. So they are charging every time they deliver full content. Again, it’s not an AI use. Right? If you’re going to a parallel or an Exa, they’re just delivering a carbon copy of content as a service for money. Right? You can hand wave all the AI inference you like. That’s not an AI inference question. That’s content as a service.
[30:49] Pete Pachal
Yeah.
[31:02] Jonathan Roberts
Priced per delivery of our content, right? Like that’s theft. Right? So that model’s pretty simple to talk about.
[31:05] Pete Pachal
Yeah, I could see like the analogy I might almost make is like at retail, you have wholesale and you’ve got the storefront, and those are different markets, even though both has a price. And to your point, you kind of need like there are only isn’t really even a standard or of for what a wholesale market looks like, you know. So that idea of like
where the per use versus per crawl is appropriate. Like in other words, you need some kind of agreements over a period of time to just get some data. You know, like I, you know, the stat that he even said about the 85%, maybe that’s true for a broad use chat bot, but maybe that’s not true for a very more specific ser service. Maybe it’s closer to 50 or even, you know, 30%. But you need you need some kind of measurable th thing to even do it and have some sort of like maybe it’s a
Do it for six months and measure and then sort of reprice like and
[32:05] Jonathan Roberts
Exactly. Look, I think we are seeing a market coming into existence in front of our eyes, right? Like the even in the last year, right? The fact that these companies are emerging, the fact that these economics are emerging. And with all two sides of markets, the market’s gonna figure out the pricing. Right? Like we can all have ideas about what it is today, but you know, give it three years from now and be like, hey, it’s very different than what we thought it was, and either it worked or it doesn’t. What I’d say is
There’s a principle to go back to on what this market has to do. If we would create new content for this market, then the pricing is right. If we would not create new content for this market, then the pricing is wrong. Right? And that is different from I think where a lot of the tech companies were at, because like, great, we’ll just take this and maybe we have to pay a bit of money to fob people off. That doesn’t solve the tech company problem, right?
Think about it like an actual supply chain, right? If we’re talking about chips and power and information, we treat information not as an asset to be mined, but as a supply chain to invest in, right? Those are two very, very, very different things, right? Take chips for a minute. When we were first creating these AI models, these LLMs, right? We did them on GPUs, right? Graphics processing chips.
Those chips were invented to play Doom and Quake, right? They were not designed to build like world spanning intelligences. Do we still have GPUs training these? No. We have custom inference chips that were designed for the task that they needed to do. What do we train them on? We train them on the wealth of human knowledge on the internet. Because that’s what was there. Is that what we’re going to need going forward? Obviously not.
Right? There is an exploding need for new information. People are asking questions of these systems that they’ve never asked before. And there’s no information to answer those questions yet. Now
[34:08] Jonathan Roberts
These tech companies can decide they’re going to synthetically create that new information. That’s called lying and making things up, right? That’s not going to solve the problem. Or you can take seriously the fact that there’s a global industry which is built around identifying needs for new information and solving it. That’s called journalism and publishing. Now you can either decide to build your own new version of solving that problem, or you can pay the people globally who already do this every day. And you can give them new ideas.
[34:14] Pete Pachal
Ha ha.
[34:38] Jonathan Roberts
Of what they need to go cover and to pay them when they cover it. The infrastructure for the supply chain exists. If you don’t fund it, it won’t. And then your AI gets worse, not better. If you fund it, your AI gets better, not worse. Like, that’s fine. That’s
[34:54] Pete Pachal
Hey, you’ve convinced me. I the but I think this is interesting because it actually touches on something that I’d love to just get your thoughts on here in I guess September, October 2026, which is the tri training versus inference. Cause like obviously like you’re talking mostly about sort of inference, which is like real-time information, bots coming out. There’s more new information, need to process it. And I do feel like while w when you sort of think about the
the long arc of y AI and the content how publishers contend with it. A couple of years ago it was all about training, cause honestly AI search engines were really not even barely there, I think perplexity and a couple other things. But now it just seems like it’s all about the inference, right? Like in just in terms of the way we use these things now. So where’s the training question now in this?
is it lower stakes? Obviously it still matters to some extent ’cause a lot of the original lawsuits were about it, but how do you guys think about that with regard to this market?
[36:02] Jonathan Roberts
Yeah, so it’s a good one to break down like what is the difference between training and inference, right? Because it can all get very kinda tied up together.
You can use a lot of content to train a very smart model, right? And that model is going to have some understanding of kind of the relationships and the like the quote unquote knowledge, right, in that training data. But when you ask it a question, it’s going to be using everything it has read from everywhere and everything it’s learned to try and provide that answer. So it’s gonna be the average, right? All of those inputs it’s picked up over all time, right?
Which is why if you ask ChatGPT or Gemini or Claude a question, but you don’t give it access to the internet to go say, do some research, it will sound very smart, but there’s no reason for it to be right. Right? Models without reference are fiction writers.
Extremely good at talking about something that sounds plausible, but with absolutely no requirement to be correct, right? And the AI industry is extremely good at euphemisms, so they refer to this as hallucination, which the rest of us would just call making shit up, right? And they’re going to get more plausible as you train them more, but they’re not going to get more right. Right? If you are able to give it an accurate piece of information for it to reason over, it stops becoming a fiction writer.
[37:09] Pete Pachal
Ha ha.
[37:24] Jonathan Roberts
And it starts becoming a researcher. These models can be exceptional researchers.
[37:27] Pete Pachal
Much more useful.
[37:30] Jonathan Roberts
They’ll still make things up around the edges, but you can go check their work because they have to cite their sources, right? And so the question is, can you make something that’s very, very smart actually useful? Well, to make it useful, you need to give it good inputs, right? Now, we know our content in the training model too, and we would say that was also wholesale theft. And think there’s a bunch of stuff that came out last week, which means that even inside these companies, this that was the framing internally. So
That’s for the course to decide. Like there’s a lot of people figuring that out. That is one of the big questions of our time.
We’re not gonna wait for the courts to figure out how to live in this economy, right? We’ll do with the inference side whilst the courts are figuring out the training side. But if somebody wants to work with us to get our content for training, we’re happy to have those conversations too. We have a big strategic partnership with OpenAI, we have a meta AI partnership, we’re deeply partnered with Microsoft. Some of those have training in them, some of them don’t, right? Different deals for different partners.
What we do know is that in all of those cases our content is incredibly valuable, right? We’re the biggest publisher in America, one of the biggest in the world, and we have a huge amount of resources to throw at creating new information and also managing the library of knowledge we have to date. Now, on the inference side, the better the input, the better the output, right? It it really is that simple. And the model matters, but not
As much as it used to for most questions, right? If you have an open weight model, if you use any one of these kind of off-the-shelf models, or just an older family model, right, and you give it great inputs, you will get better answers than the bleeding edge frontier model with no inputs. Right? Which means the inputs have more impact on the quality of the answer than the model itself.
[39:03] Pete Pachal
Right.
[39:24] Jonathan Roberts
Now you’re spending hundreds of billions, if not trillions, of dollars on making better models, and you’re spending effectively zero dollars on better inputs. Well, if the biggest impact on the quality of your product is the inputs, you’re spending your money in the wrong place.
[39:41] Pete Pachal
Gotcha. So yeah, like training obviously still important, still is a part of the table sticks, but it is like the action is where we’re we’re in friends. Cool. Before you go, I wanted to get your take on some newer stuff in the AI world. So time famously in the summer started partnering with Mobian to be essentially serve ads to bots and you know essentially
[39:46] Jonathan Roberts
Table stakes.
[40:09] Pete Pachal
Put a marked down ad on a markdown page that then gets scraped and then you know this is potentially the beginnings of sort of a new kind of ad market it that’s all sort of machine driven. It’s very early days, of course. There’s only a few companies kind of doing this kind of ad tech. And but they’re there’s there’s it’s an interesting idea. Obviously just to fill in people, like perplexity came out shortly after, said this is probably a no-no
no one else has sort of come out and we’re sort of seeing how it goes. But it I find it very interesting, this whole this whole idea of basically a new a new kind of revenue channel potentially in the whole idea of content getting scraped and putting it out there. What do you think of this? Is people doing anything in this area? Is it just an experiment? where what do you think of it currently and where do you think it could go?
[41:09] Jonathan Roberts
So I think there’s there’s two sides to this coin which it actually speaks exactly to what we were just talking about. AI is now an intermediary between people and the information they’re trying to get to, right? So we need to deal with these systems, right? Blocking but also enabling, right? Carrot and the stick, to make sure that we can navigate to the consumer the information they need. Again, like the AI is not the end consumer.
The AI is just a tool that a consumer is using to get the information we want them to get to, right? So we want that new economy to work, we want it to figure out how to play in it, we want it to figure out how to make it better, we want to invest into it. If you’re a brand, you have the same problem. There’s now an intermediary between you and your customer, right? And that is some AI layer, whether it’s agents or chatbot or whatever it is, right? So it’s the same problem, right? Brands need these systems.
To accurately represent them. Right. And at the moment, a lot of the time, these systems do not accurately represent brands. There’s an amazing BBC and European broadcaster and public service broadcaster study which showed that the s 60 to 80% of the time these systems make factual errors when they represent an answer. And if when they make an error, consumers blame the source, not the system. Right? So
A brand is being misrepresented to consumers and consumers will blame the brand. Right? That is a problem. That’s a huge problem. Right? That’s a brand safety problem. That’s a brand representation problem.
What’s also happening is that we all said we wanted to automate away all of our chores. Right? Like this is the kind of utopian idea. Like, great, the robot can do the chore. What happened is they took it too far and they automated the choice. Right? Automate my chore, but do not automate my choice. But these bots are taking both the chore and the choice. Right? And again, people like convenience. Everyone likes convenience. Like, do I want to make all these choices every day?
[43:05] Pete Pachal
Right.
[43:17] Jonathan Roberts
Really like convenience does often win, right?
[43:20] Pete Pachal
Right. Until until there’s a bad choice made on your behalf.
[43:24] Jonathan Roberts
Well, if you’re trying to influence a lot of people, previously you would try to influence all those individual people. Now you’re heading into a world where you get to influence three people Claude, ChatGPT, and Gemini, right? that’s very dangerous. Right? Because these systems are all gonna make choices the same way. Right? And so that removes all this diversification of choice and starts making one choice, right? That’s that is the nightmare for a marketer.
If I say what’s the best hotel in New York for my family coming into town, I’m not getting a choose your and adventure of like twenty links on a search page where I can pick, but I know their price point, but I know what they like. I will make choices because of my own knowledge that the search page doesn’t know. Search is very humble, right? Search allows you to pick your path. Chatbots are arrogant. They decide it knows for you already and it makes a choice. Right? That’s a huge problem for marketers.
and so they want to influence what the answer is and where it’s coming from. Now, there’s a couple of different ways to do this. You can either try and game the system and manipulate the algorithm, right? Or you can produce trusted content transparently that provides accurate information and fixes some latent biases and errors that they have, right?
Perplexity claims times is the first, right? The perplexity statement was this is deceptive, which looking at perplexity’s past behavior, I would call ironic. But we’ve been through this before, right? These are the early days of search. It is not a question about whether or not brands need to be influen need to be able to influence consumer choice, right? That’s their job. Right? And if we’ve taken the choice away from the human, then you have to influence the system.
[45:09] Pete Pachal
Right, sure.
[45:16] Jonathan Roberts
I would argue, why the hell are we taking the choice away from the human? That’s a bad idea, right? Like, yeah, take away the common garden baseline commodity choices, fine. But leave people with the in choice the choices that matter. And those are all being taken away at the moment. And that’s not an AI problem. That’s a UX design problem. That’s a product decision. That is a product choice. This is no this isn’t some inevitable outcome of AI. That’s a product person deciding to remove consumer choice. Right? Now think about that for
[45:25] Pete Pachal
Mm-hmm.
[45:46] Jonathan Roberts
You know, what brand of toilet paper to buy, one thing. Think about it for an election, and you’re in a totally different. Think about it in a health choice where rather than that person knowing some things about themselves that they didn’t type and then being able to choose their own search answers because that reflects them, or the chatbot assumes they know everything about you and gives you average health advice but not accurate to you and you’re having a bad health outcome, right?
[45:54] Pete Pachal
Right. Yeah.
[46:06] Pete Pachal
Mm.
[46:15] Jonathan Roberts
If we are automating choices, there are a lot of bad outcomes for this, and it’s a lot worse than do you stick a markdown ad in front of a chat bar, right? This is do we as a society want to be in control of our choices?
[46:31] Pete Pachal
Right. Okay. So it gets complicated quickly when you go down all these paths. But generally do you think that because of the way these systems are evolving, that this is a thing that like, you know, like to your point about minimizing choice or steering people, it if things continue as they are, I guess depending on how the other platforms come in, it’s do you think this could be a viable path for revenue?
for publishers.
[47:02] Jonathan Roberts
There’s obviously a lot of dollars stacked against trying to solve this problem, and publishers are the source of trusted answers. So there’s a real partnership to be done there. If you want to come and work with us, we have a whole large language model visibility program that we work on that we work with brands on. It’s been very successful. It’s running at a nice clip, and we work with all of our trusted partners and the platforms that we partner with.
[47:32] Pete Pachal
Nice.
[47:33] Jonathan Roberts
Well we will work with you to make sure that we have the right content in partnership with you that reflects your brand properly, and we will clearly declare it as a piece of content that’s been written in partnership. And we’ll do the research to figure out where the gaps are, the biases you need to correct for, the things that aren’t being represented accurately, then we’ll work with you to fix it, whether it’s on our platform, on your platform, or across the information ecosystem, right? We’re huge. Sure.
[47:57] Pete Pachal
So it sounds like you’re doing a version of this and then and you’re you’re able to measure sort of outcomes from
[48:01] Jonathan Roberts
Well, our brands are the most trusted sources for all these models, right? So we are a huge footprint about all things these models ingest and care about, which is why we have the partnerships we have.
[48:12] Pete Pachal
I definitely would love to go down that rabbit hole even further with you. But we’re we’re kind of up on time. I really gotta get to I what you the basically the final question, which is what are what are you worried about in the world of AI and what is excites you? What are you optimistic about?
[48:27] Jonathan Roberts
I’ll go at the exciting piece first.
I think it’s clearer than ever that we need a lot more information production, a lot more knowledge creation, a lot more journalism in the future to make these products good. Right? That was, I would say, an edge case belief a year ago. It is blindingly obvious this year, right? The VC dollars are flowing into this when the buy side is buying this way. Like there is definitely a market for an explosion in need for new knowledge, need for new information, need for trusted sources, right? That
Is very, very good for a future publishing industry if we wire that demand up to the people creating it. Right? So this is a knowledge revolution. We should live in a world where knowledge creators are in a booming era. We’re not there yet. Obviously, we have not connected the demand to the supply yet. The thing that makes me concerned is we’ll take too long to figure that out. Too many players will have too much vested interest in not paying for their raw materials and keep stealing the fuel.
And then there’ll be no fuel. And then we end up in this kind of mediocre world full of AI slop and we ask ourselves, How do we get here? And
In the next couple of years, we’re gonna find out which of those futures we end up in, but we all get the choice to help determine which of those two futures we end up in. And step one is the best way to stop people selling your content for free is stop them taking it for free. If you’re not blocking people from taking your content, you’re leaving your door wide open whilst you’re getting robbed. Step two is figure out how to actually partner with the people who can be good actors, because there are many.
[50:12] Jonathan Roberts
And support them in what they’re creating as their create a market which will actually pay for the future information revolution.
[50:19] Pete Pachal
Nice. We’ll leave it there. Make good choices and make sure they’re visible. Jonathan Roberts, I really appreciate you coming by and sharing your thoughts on all this. This is a awesome. Thank you.
[50:22] Jonathan Roberts
Yeah.
[50:30] Jonathan Roberts
Always a pleasure. I’m sure in the next year and a half it’ll sound exactly the same. The world is changing fast, so I’m sure that’s not true. Great to see you. Thank you very much. Bye for now.
[50:34] Pete Pachal
We’ll check in then. Take care.
