By The Copilot

Bots now make up about half of all web traffic. Are they a threat to block or an audience to win? 

For most of the web’s history, the deal was simple. Search engines pointed people to websites, and those visits paid for the content through advertising and subscriptions. AI is rewriting that arrangement, and the clearest place to see it is in who, or what, is showing up at the door.

Akamai says it now handles more than 150 billion bot requests a day, with AI bot traffic climbing more than 300% year over year. Bots account for roughly half of all internet traffic. On this episode of The Media Copilot podcast, Pete Pachal talks with two Akamai executives who sit on opposite sides of that shift: Kim Salem-Jackson, the company’s chief marketing officer, and Patrick Sullivan, its CTO of security strategy. Their jobs once had little to do with each other. Now they are working the same problem from two directions.

For Sullivan, bots have been a security headache for more than a decade. What changed is that the most valuable visitor to a site is now also a bot: the retrieval and training crawlers that feed large language models. “The VIP visitor to the website are the various bots that really make LLMs go,” he says. Detection and blocking are no longer the point. Akamai keeps dozens of categories of bots and a menu of responses for each.

For Salem-Jackson, the reframe is sharper. She treats LLMs as a new audience, each with its own personality, the way a marketer treats different buyers. “The bots are my new customer,” she says. She monitors bot volume by the hour, rolls out a welcome mat, and watches whether her team’s work on AI visibility pulls more crawlers in.

The catch is that the same bot looks like an opportunity to one business and a threat to another. A training crawler can compress a publisher’s entire site, hand it to a model, and never send a reader back. Sullivan puts the ratio of training visits to human visits at tens of thousands to one. For a company like Akamai that wants to show up in AI answers, that crawler gets cookies and milk. For a publisher whose business runs on traffic, the same crawler is something to block or charge for.

That’s where the conversation gets useful for media people. Akamai offers pay-per-click tools, but Sullivan is candid that those models are early and that content licensing deals remain more common. The technology is the easy part. The hard part is the business and legal call about what to allow, and who can enforce it. Unlike Cloudflare, which has taken a loud public stance on publisher control, Akamai casts itself as “Switzerland,” a technology enabler that leaves the decision to the publisher.

Then there’s the part every publisher can act on now. Salem-Jackson says an average webpage runs about 200,000 tokens, but a model reads only around 10,000. Roughly 1% of your site actually gets consumed. Anything important that sits below that budget is invisible. Her fix was a separate “bot site” that serves crawlers stripped-down, high-value content, which she credits for an 85% lift in citations. Great content is not enough if the machine cannot read it.

Key takeaways

 AI bot traffic is up more than 300% a year, and bots now make up about half of all web traffic

• For some sites, the most valuable visitor may now a bot, not a person

• The difference between training bots, retrieval bots, and malicious bots

• Marketers are starting to treat LLMs as an audience to court, each with its own personality

• The same crawler can be an asset or a threat depending on whether your business runs on visibility or on traffic

• Pay-per-click for bots exists, but licensing deals are still the more common path, and enforcement rests with publishers rather than the CDN

• A model reads only about 1% of a webpage, so anything below the token budget is invisible to AI

• A separate bot-optimized site can raise AI citations sharply, which Akamai puts at 85%

• Wikipedia, LinkedIn, YouTube and Reddit carry outsized weight in what LLMs cite

🔗 About the 👤 Guests

Kim Salem Jackson
LinkedIn: https://www.linkedin.com/in/kimsalemjackson/

Patrick Sullivan
LinkedIn: https://www.linkedin.com/in/patricksully 

Akamai Technologies
https://www.akamai.com

About the show:

To explore more conversations like this and see what’s new, visit the Media Copilot website at mediacopilot.ai. You’ll find new episodes, expanded resources, and tools designed for journalists, communicators, and media leaders navigating the fast-changing world of AI. It’s the home base for everything Media Copilot and it’s just getting started.

Enjoyed this episode?

Subscribe to The Media Copilot on Substack, Apple Podcasts, Spotify, or your favorite app. On YouTube? Tap the Like button and Subscribe to the YouTube channel. For more AI tools and resources built for media professionals, visit mediacopilot.ai.

Produced by Pete Pachal and Executive Producer Michele Musso
Edited by the Musso Media Team 

Music: “Favorite” by Alexander Nakarada, licensed under CC BY 4.0

All rights reserved. © AnyWho Media 2026

Transcript

Pete Pachal: Hi, welcome to the Media Copilot. It’s a podcast about how AI is changing media, news, and communication. My name’s Pete Paschel, and I covered tech for a long time as a journalist. And now I have deep conversations with the media people, the builders, and the creators who are all answering the question how do we get information in the future? And how will that change the jobs and the industries whose business is information, especially media?

Kim: Okay.

Pete Pachal: For most of the web’s history, there was a fairly straightforward exchange. Search engines indexed information, sent people to websites, and those visits helped support advertising, subscriptions, and the creation of more information. AI is changing that exchange. Bots and agents can now read a site, summarize its work, recommend its products, answer questions based on its information, and sometimes even act on a user’s behalf. The original source may receive a citation, a smaller number of visitors, or really no measurable benefit at all. That puts companies such as Akamai in a powerful position. The infrastructure sitting between a website and the rest of the internet can identify automated visitors, can decide what gets through, enforce access rules, and potentially create new systems for licensing and payment. Akamai says it processes more than 150 billion bot requests each day and has recorded more than a 300% annual increase in AI bot traffic. It is developing tools that let companies block, permit, or verify, or monetize AI access while also helping brands improve how they appear in AI generated answers. My guests today work on opposite but increasingly connected sides of that problem. Kim Salem Jackson is Akamai’s executive vice president and chief marketing officer. She leads the company’s global marketing operation and has overseen its work on measuring and improving brand visibility inside AI platforms. Patrick Sullivan is Akamai’s vice president and CTO of Security Strategy. His work includes bot management, how to identify agents, edge security, and the systems that could allow content owners to control or charge for automated access. Kim, Patrick, welcome to the Media Copilot.

Kim: Thanks for having us, Pete. We’re thrilled to be here.

Pete Pachal: Nice. Good to see you both. All right. So before we get into all the tech, I would love to understand your both your two areas a little bit line a little better. I know I summarized them just there. but it I want to understand them and also how they converge around the problems I outlined. So Kim, maybe why don’t we start with you and you can begin from the marketing side and then Patrick, why don’t you take over and explain how it looks from a security perspective.

Kim: Sure. Well, you know, as a CMO I think about two main things. I think about our our brand and awareness and I think about driving demand and revenue for sales. And obviously AI has completely changed the paradigm on both sides of the coin about what I get up every morning thinking about. And the partnership between the business and IT, and in this case working with Patrick, has never been stronger, right? As AI becomes and AI bots become one of our most important customers and our VRP visitors. So that’s a little bit about Macro my role and how I think about the whole world. changing.

Pete Pachal: Patrick, how’s it look from your end?

Patrick Sullivan: Yeah, absolutely. So so I, you know, I think on the security side, the application security team has been handed the challenge of managing bots for more than a decade. you know, that’s an area that you know, I’ve been working with some of the world’s largest brands to help deal with you know, some of the threats, including sort of the relentless testing of compromised credentials on websites. That’s been sort of the number one way websites get breached for you know, most years out of the last decade. so the the security team was dealt that challenge of, you know, can you identify the bots, categorize the bots, optimize the way that you respond to the bots, and then provide monitoring and and analytics. so that, you know, we continue to see fraudsters, you know, leveraging bots, but as Kim said, on the other end of the spectrum, Now the VIP visitor to the website are the various bots that that really make LMs go, right? The retrieval bots, the training bots, et cetera. so that is now the the most important visitor.

Pete Pachal: So it sounds like from your perspective, the change isn’t just volume. It’s that the nature of the bots is different. And whereas before I imagine it wasn’t un universal, but generally like if you saw a bot, it was probably not doing good things. And then and now it’s like, well, half the internet’s bot traffic because everyone’s using AI. So it’s a sort of a difference in kind as well as a difference in volume.

Kim: And you as a marketer, I always kept an eye on the bots. Obviously, we’re Akamai and we use our own technology, but now I’m obsessed with the bots and I want to understand good bots versus bad of the good bots, where are they coming from? As Patrick said, what are they doing? You know, how many can I let in without my security team getting nervous that we’re, you know. letting in the bad bots. So it’s interesting how even as a marketer, my obsession has changed and, you know, I’m monitoring it all the time. Whereas before I’d probably look at it, you know, every six months just just to kind of understand the ebb and flow of them.

Pete Pachal: So imagine there must be like good bots, bad bots, and sort of gray bots, you know? Like I how do you classify these? This is probably more your area, Patrick, but I mean like I guess in terms of how you’re applying the filter. And then Kim, how are you interpreting what’s getting through that filter? Why don’t we start with you, Patrick?

Patrick Sullivan: Yeah, Pete, you nailed it. I mean, there’s a whole kaleidoscope of of bots, right? You know, starting with classically the the the sacrosanct bot that you don’t want to mess with is the Google bot and Bing Bot. You know, you wanted to make sure that that you showed up on search, you know, historically. then you would have some partner bots, you move your way down the spectrum. Maybe there are some aggregators and scrapers that you know, it’s it’s okay if they’re coming through as long as they don’t start to become excessive or cause issues to the websites. And then, you know, you move all the way over to the more parasitic bots that are fraudsters and competitors scraping your website. So there’s always been this continuum. So we’ve always thought about detect the bots, which is not trivial, categorize the bots. You know, we have dozens of categories for the bots and it’s not about blocking the bots, it’s about having you know, a whole menu of responses. And then also you want to make sure that you have your monitoring and analytics so you understand the implications of what you’re doing. But I would say things have radically changed as I’m sure Kim’s about to tell you, you know, that optimize verb, we’ve added some things around making sure that the AI bots in particular get exactly what they want. And there’s a feedback loop there with the monitoring and analytics.

Pete Pachal: Well, come back to that thought about you know, th not just blocking. I think you it’s it’s can be blocking, not just not always. I I think is what you meant. So there’s more subtlety there. But Kim, like obviously you’re thinking this not just in terms of bots, but like audience, right? And who who are these bots representing? What is as as sort of these bots get through to sites, like how do you how do you interpret that?

Kim: Yeah, I mean, Pete, you nailed it. To me, the bots are my new customer, right? I think of the LLMs as a new target audience for us, which marketing has done since the beginning of time. And then, you know, I’m also thinking about, okay. Each bot has a or each LLM has a different personality, just like each buyer in marketing has a different personality, business or IT. So my team is not only working with Patrick and understanding what the bots are, but you know, where they’re coming from, what they’re scraping, what they’re doing, and how I can more precisely market to them to align to their needs. So I literally think of them as my new target audience and I’m trying to understand their behavior, retrieval. training. I like the retrieval ones because I’m hoping they’re delivering information about Akamai that’s a signal to buying. That’s my favorite behavior. I like to look at the volumes and as we kind of lean into some of our GEO work is that having an immediate correlation to more bots coming to our site if we’re as we’re trying different tactics. So I am constantly monitoring that bots. I have a big welcome sign to let them in and you know I’m sure we’ll get into what we’re doing with our website and The bot site versus the human site, but yeah, I one hundred percent think of them akin to a human. And that the way I would target business buyers versus IT buyers, I just think of those LLMs in exactly the same.

Patrick Sullivan: Yeah.

Pete Pachal: Right. So some people in my audience will definitely understand that the whole idea of the welcome ad and sort of making sure the bot the good bots, I guess as we’re sort of throwing a lot of like judgmental terms here, but like the bots that are having a positive outcome for whatever the business goal of the site is, they would want to well sort of lay out the right carpet and make sure they have a good experience. So that’s a good chunk of my audience or comms marketing, but then there’s another chunk of my audience, which is obviously publishing journalists, editors, and they’re like, Well, wait a minute, like we are

Kim: Come on in. Yeah.

Kim: Exactly.

Pete Pachal: business model is all based on advertising, subscriptions, and other things that depend on traffic. without some kind of compensation system in place, I don’t know if I really even want these bots in place. So let’s k stay with you for a second, Kim, as you as you sort of think about that in terms of like who might be on the other end of whatever Akamai is managing as a CDN and what might be available to them. And and how how that might that like in other words, they might be applying a different filter to to the things you’re talking about.

Kim: Sure, and I’ll I’ll tee it up and then I’ll hand her to Patrick. So obviously we have a breadth of customers and some are like me, you know, that it just wanna put out that welcome at and others need to monetize their site, right? And and really kind of allow that pay per click. And so based on, you know, step one for us at Hakamai is really to understand your business objectives, what role the bots play. in your business model and how best you want to treat them and calibrate it over time. And so you know that’s really where we start every conversation with our customer is understanding their business objectives, their business model and how we can meet them where they are and put the right monitoring and restrictions in place based on that. And Patrick, I know you’ve worked a lot with media customers and been sitting with them. Why don’t you share a little bit more about how we put that structure in place for them?

Pete Pachal: Right. Before before I wanna I definitely want to hear from you next, Patrick, on this and the structure you have in place. But I up until now I think we’ve been sort of talking loosely because we’re just having a chat here about like good and bad bots. But honestly, I loved it if you could get even a little more clinical about it and think about like the different types of bots. So it’s kind of like like the way I think most of my audience and I understand them is like there’s essentially three main types, you know, you have training bots, search bots, agent bots. You might think of it have a different taxonomy in terms of what’s relevant and how you filter, but sort of like as you think about like the different reasons a publisher might want to block or allow through, I think understanding those subtleties would be really helpful to people listening. So please let me know like how the your customers approach this and what you find is the best system that tends to work.

Patrick Sullivan: Yeah, Kim, maybe I’ll take the first crack at this. Yeah. So Pete, there are many, many more categories of bots there, but I think you’re right. We should we should drill in, right? There’s a whole subset that we call the AI bots. And Kim and I both have been talking about training bots, for example. so to to really be specific, what the the role of those bots is on behalf of the LLMs, they’re all visiting websites, you know, ingesting that information and feeding that back into the LLM such that, you know, when an when an agent or somebody interfaces with a chat, they have learned you know from across the web and they’re able to incorporate that knowledge that they derived, you know, from the training bots into the response from the LLM. but but Pete, you nailed that. There is a massive difference in sort of the way people think about those training bots. You know, Kim cannot get enough, right? Like every time a training bot’s coming, you know, she’s you know putting out cookies and and milk, you know, come back, invite your friends. Because you know, because the the really for her, I’m sure she’ll get into the metrics that she tracks, but the more that they come and the more that

Pete Pachal: Literal and virtual cookies, yes, I get it.

Kim: Exactly.

Patrick Sullivan: the LLMs learn about the content on the website, the more that people researching, you know, topics that are relevant, you know, such as, you know, how do we protect ourselves from the latest Frontier LLM security threats or other topics? If those training bots are coming to Akamai and and ingesting our content effectively, we show up for those users. But Pete, to your point, there are other people who really thrive their their whole economics are around intellectual property, And making sure that that’s monetized effectively. And they look at that exact same bot with the exact opposite response. They don’t want that bot coming in for no monetary value, making off with that content and then monetizing it without them, right? So there, you know, the technology is frankly much easier than the legal and business, you know, on this side of the equation. We can do the same thing, we can detect that bot. Categorize that bot. And then when we get to the optimize, rather than the optimize response that Ken will detail in a minute, there, you know, they want to maybe block the bot. Maybe they’ll they want to have a specific API because they’ve structured a financial agreement, you know, where they’re a specific LLM has has structured a deal with a content provider. So rather than crawling, maybe they have a dedicated API. or maybe they’re paying per click. those things are pretty easy for us to instrument from a technology perspective. Honestly, the hard part there are the the lawyers and the business people that have to make those decisions. because it’s pretty easy to say we’re going to block this LLM. The business implications are are far more complex. But yeah, absolutely different people look at the exact same bot from a hundred and eighty degree spectrum. and and we can just help them implement that policy, right? That’s we give them the analytics and then when we get to that optimize, it it is customized to the customer and their business model.

Pete Pachal: So Kim, again, the the the idea of these different kinds of audiences and customers that you have, you know, it seems to be that even within the publishing world, like even if you do have content that is IP invaluable to you, you it’s not a blanket thing typically, right? That you just you want to block everything or protect everything. You might want to like have some content visible and some others. but I know Patrick, you talked about training. Is training really the the main thing right now? I do feel like for a lot of publishers, it’s more about rag these days because they’re constantly publishing content and I don’t know, like so maybe they block training, maybe they don’t, but it’s also about like the stuff that is appearing in people’s answers without the need for them to go to the site anymore. I don’t know if that if the conversation is encompassing all the bots or or if training’s still a part of it, but like I guess what are some of the the subtleties that people can sort of put into this process given the tools that you offer?

Patrick Sullivan: Yeah, mean I I I think it comes back to you know, as as Kim detailed, the the shift in the web, right? you know, it used to be you would go to a search page, click on a link in Google, and then that would navigate you to the web page. These days, if that training bot comes through there and it can you know, compress the information from the website and then incorporate that into the LLM, the the ratio of you know that that training visit to Actual visitors going to the LLM is brutal, right? It could be tens of thousands to one. So one training request comes in, it informs the LLM, and then tens of thousands of of visitors could just, you know, go into their chat interface with their favorite LLM, and there are no subsequent calls back to the website that can be monetized, right? So that the the economics for somebody trying to monetize site visits, can be damaged very, very significantly. And again, for that reason, things like retrieval and and training bots are are treated pretty harshly.

Pete Pachal: Yeah, but I would guess I would I would ask like is is there guidance that you give to publishers on like, well you might want to block training and search but not agents or or all of it? Or is there are there reasons, common reasons you might want to allow one and not the other?

Patrick Sullivan: Yeah, a lot of it it there. I mean, if if your model is strictly about monetizing, you know, I think it comes down to can you strike a content deal? you know, it does blocking some of these bots give you more leverage to to structure that deal. It you know, it’s the technology is is in support of the business arrangement.

Pete Pachal: Well what about like these things where you can play for the usage or the crawls and the sort of like automatic payment or you know, paywalling, I guess, of content from the bot internet? Is there what is what does Akamai offer in that?

Patrick Sullivan: Sure. s so the we do offer, you know, what we call pay-per-click. You know, in that model you would look to monetize as a content provider, you know, every time that that there is a a click within the LLM. I would say in general across the industry, though those models are still you know, emergent. I think they’re still starting to pick up adoption, but very early days there. I I think probably more likely scenario are the the content licensing deals where, you know, again, an LLM would get together with an intellectual property holder and and they would structure a deal for exclusive or or non exclusive use of that content. That’s probably more common than the pay per click model on the internet today.

Pete Pachal: Right. And so you know, I guess it sort of comes down to why, right? Which is to say like I think a lot of AI companies or those that are in the business of I guess you call information brokering, i they there there’s easy ways to get the information even with bot blocking in place, whatever whatever that may be. So you know, it you might be aware that you know Cloudflare’s taking a highly public position on publisher control. presented itself as kind of an advocate for for changing the economics of this. the AI crawling that is. And it’s it’s emphasized giving customers a lot of options. like how how do you how do you how does Akamai sort of see its role? does it have a similar position? Is it a little more is it a different audience, different different philosophical stance? maybe Kim you want to take this one and

Patrick Sullivan: Yeah.

Pete Pachal: And maybe Patrick let me know how it sort of plays out in in the in the technical side.

Kim: Yeah, I mean our our view is again we want to meet our customers where they are based on their business objectives. So you can take a hard and fast stance or or you can adjust. I I was just on a call with a retail customer who views, you know, one of the LLMs as their competitor and they basically wanted to take a really strict stance on that and almost block them entirely. So some of our competitors, you know, take one direction, others are a little more loose, you know, Akamai’s a little more like the Switzerland. We basically again wanna understand your model and help you adjust and and block. Patrick, anything you wanna add? I know you speak to a lot of customers on this top.

Patrick Sullivan: Yeah, Pete, I mean, I would say, you know, if you look at sort of the the delivery engine for the the media industry, you know, Akamai is the top of the list there. on the bot side, you know, we have a a menu of those optimizations that’s broader than anybody. so I would say, you know, there certainly have been some announcements in the industry about changing the fundamental economics of pay-per-click. I I you know, I would ask you to follow up with with some of those folks on the adoption there. but I would say in general across the industry, you know, maybe some of the the announcements have not been followed by quite as much adoption and major shifts to the the funding of the of the internet may not have followed up, you know, with some of the pronouncements that that have been made, but that’s probably for somebody else to to respond to.

Pete Pachal: Well, I guess I guess what I was getting at is that there there’s an enforceability to this that that seems like is relevant, right? Which is to say, like if you’re a major CDN like a Cloudflare or an Akamai, that gives you real influence in the industry, Cloudflare is obviously using that to adopt an advocacy position. And is there is there w I guess would the industry and the things you’re trying to do to help your customers, particularly ones that want a block, would the Would that be helped by either new standards or new maybe even regulatory practices? So for example, what I I know this has been proposed by Tolbitt, who I believe I I’m not sure if you guys are doing things with them. Yeah. So the the in terms of bot identification, that seems to be like, at least from my conversations with them, that’s a good place to start. In other words, requiring that there’s a transparency to what the bots.

Patrick Sullivan: We do. Yeah, we have yeah, yeah.

Pete Pachal: function is, which I think most good players do. But how enforceable is that if players like Cloudflare or Akamai aren’t sort of actively really encouraging that? And I guess I know you both do. It’s just that where where to to what extent, I guess, might be the the the question.

Patrick Sullivan: Yeah. So so I think if we if we follow that through, you know, the the role that Akamai would play in pay-per-click is the technology enabler, right? So if a publisher says we want to move forward with pay-per-click, and the other side of that is, you know, prevent anybody who’s not participating as an AI bot. If they’re not participating in pay-per-click, then we’re gonna block them. they would have to make that decision, right? That that decision rests with the publishers. we’re sort of the enforcement mechanism there, the detection and and technology decision. so it’s really not up to the the bot provider or the CDN to make that decision. That’s sort of the technology enabler. The business owner at a major publisher would have to say, We’re willing to deal with all of the repercussions of blocking LLMs that are not participating in Pay-PerClick. And I think that’s the rub and and sort of maybe what’s driving the current level of adoption of Pay-PerClick. But the the technology is there for publishers that want to adopt it.

Pete Pachal: Okay. So just to probably my last question on this just to say that I think it is like the security framework taking the sort of bot identification, it’s like identifying the agent, like who’s behind like who’s operating it. Like I don’t know if this is there’s an equivalent KYC with with bots or anything. And then just what the purpose it is doing, right? So like that that to me Sounds it would fall on the C DN, that kind of idea. Like just what is the bot and we want to be clear on what it is. So I guess my question would be, is there is there a minimum credential, I guess, for a legit AI agent? And I without getting too technical, what would that be?

Patrick Sullivan: Yeah, so so their AI agent is also a very broad category. So if you think about that, you know, we have you know a whole we published a security framework for you know agentic bots. So we participate in partnerships with major credit card companies, you know, major identified agent platforms. So we we have the ability to understand the trust level of that agent platform, and then we also

Pete Pachal: True, sure.

Patrick Sullivan: Look at the identity behind that, behind that agent. Like, you know, who’s the human identity? And then, you know, there’s various anti-fraud technologies we put in place there. But I guess to bring it back to kind of the framework, we do the detection, we do the categorization. A customer would choose which specific optimizations are right for their business. You know, pay-per-click, block. There are specific optimizations that you want to put in place. if you’re you’re Cam and you want to make sure that you show up on the LLMs. So I think we’ll talk about that in a minute. But there’s a whole menu there and that’s really at the discretion of the the content owner. and publishers respond very, very differently than a commerce website or a travel website or a bank. many organizations now, their number one visitor. the reason you build a website is to attract that AI bot so that you show up and inform the LLM.

Pete Pachal: Mm.

Patrick Sullivan: about the purpose of your business. But it’s very, very flexible, up to the discretion of the the policy of the business. Cool.

Pete Pachal: Cool. Well, let’s switch tracks to that that thing I know Kim’s excited to talk about, which is the visibility in AI answers. And, you know, assuming again you you want to be, that what what is offered and how you guys sort of help enable that. So I was looking the looking that this up, looking this up, and it says, I guess you have an AI brand presence product that it produced an 85% increase in citations. And a 364% increase in non-branded searches, 130-30% increase in child, in other words, up across the board compared with various competitors. So I’m gonna take those those figures as as as as you as you as I saw them. but how how did how do you achieve this? How do you how do you change what do you change in the back end? What are you seeing that other people don’t? What is how does your position as a CDN give you this kind of advantage so that brands can get these kind of results?

Kim: Sure. well there’s kind of two main components to what we look at. One is understanding as you you’ve listed off your visibility within the LLMs, your sentiment, your citations. And then the other one is the action you take to improve it. Right. And that’s where Alchemy Secret Sauce being, you know, a CDN, being in security and being able to deliver that website for the bot comes into play. So gosh, back in the fall of twenty twenty four, I’m losing years Pete, you know, we saw this yeah, we we can’t believe we’ve been doing it this long, we saw the sea change happening in the market and we said, hey, we’ve got to get ahead of

Pete Pachal: Mm-hmm. Yeah.

Pete Pachal: We are.

Kim: This right. And so we started with the visibility journey, which is understanding where we stood vis-a-vis these LLMs on those three KPIs I mentioned. And then the question I always say: data insight action. Okay, now that we understand our standing, what do we do about it? And that’s when we reimagined our org, how we think about content, how we think about content placement. and we started working really, really hard to achieve those stats that you just read off. But what happened in parallel is we were focusing on understanding our visibility, reworking our content engine, reorging marketing, Patrick was working with one of our partners to kind of say, Hey, how can we better serve up information to these bots so they can consume it quicker, more economically? And that’s where it became kind of this perfect marriage of what we were doing and we saw so much success on the Akamai side. And given, you know, who we are as a company, we decided to repackage it and to sell it to all of our customers. So Akamai brand presence gives you those two components. One, helping business leaders and IT leaders, we talked to both, understand their visibility vis-a-vis the LLMs, and then more importantly, give you the ability, it’s I call it more the easier button. to deliver a website specifically for bots. And the second our website for bots went live in November, we achieved incredible results. And you know, those are things the board cares about. I talk to our board about those every single quarter when I do a presentation. So we’re super excited to have to drink our own champagne as I like to say and offer this to to the market.

Pete Pachal: Nice. What are some fundamentals? Like basically what are some basic things either companies, sites, brands aren’t doing or are doing badly or wrong that you’ve found they that they can change, that they can start seeing better results. what’s the I guess what’s the lowest hanging fruit?

Kim: Yeah. Yeah, I mean f first of all, a LLMs consume only one percent of your website, right? So if you make assumption that just because you have great content, build and those LLMs will come, you’re absolutely wrong. If you don’t figure out how to optimize your site to make the consumption quicker and easier, you’re dead in the water. You can have the best content strategy, you can reorg everything, but at the end of the day, it’s about the LLMs being able to quickly consume. your information. And that was really, I think, the secret sauce. So understanding your visibility, which is probably very low if you don’t have a bot site. And then quickly standing up a site for those LLMs so you can get more of your content consumed quickly and at a better economic price.

Pete Pachal: So you mentioned efficiency there, and I understand that like the amount of data that the A systems that they need to process is you you reduce that considerably from when sort of you were you were first studying this. And how do you ensure that like I guess essentially that key th key data hasn’t been lost and it’s still also like consistent with what people see? yeah, how is that is that just technical stuff that d we don’t have to worry about, or is it like Like hi that just seems like an a pretty impress is it just something else only a C DN could do? No.

Kim: well it’s by design. So the average website consumes about 200,000 tokens and AI reads about 10,000. And so, you know, what we did is prioritized our highest value content and make sure that was on our bot site as quick as as possible. And then AI LLMs, they love breadth of of content, they love recency of content, and they like the distribution of consistent content everywhere. And so underpinning having the right tools and the right site. You also have the right, you need to have the right ecosystem under you to make sure you can constantly feed those LLMs in the way in which they want to consume your information. I think the biggest mistake people make is they think you can do one thing or the other, but everything has to work in concert. It’s an entire ecosystem it takes to feed those LLMs, right? From understanding your visibility. serving up the right content in the right places too, because a big aha we had was understanding what sources the LLMs biased. and you know, there were sites I could have cared less about or even Wikipedia I haven’t thought about in 15 years. Now I love Wikipedia because the LLMs love Wikipedia. So I think understanding your stance, understanding the placement of your content, having a content machine, and then again having that brand optimized tool.

Pete Pachal: Yeah.

Kim: to serve up the content via bot site to those LLMs is kind of the secret sauce.

Pete Pachal: Yeah, it does seem like there’s at least kind of a a set of sites. I I see the same four cited all the time as sort of the main things that have outsized influence in LLMs, and that’s like you said, Wikipedia. There’s LinkedIn, there’s YouTube, there’s Reddit. Those seem like the big four now. Is that just kind of a a natural consequence of having essentially user generated, big user generated repositories? Do you think this is something that’s gonna continue? or do you Do you feel like that might get eventually flattened out as the AI web evolves?

Kim: Yeah, I think Pete, it’s all of the above you nailed it. I think the LLMs had an affinity to those sites because they were, you know, raw, structured, plain content, not a lot of text, not a lot of fluff. Now the world understands what the LLMs like, which is why, you know, we launched a site, a bot site for LLMs and we offer it to our customers. But I think the only constant is change in this industry. And so literally every day we’re studying the market. We’re studying the behaviors of the LLMs and we’re trying to ensure that we meet those LLMs where they are with again the right content and the right format in the right place at the right time. And you know, our brand presence tool I think gives people a leg up in achieving that quicker and easier.

Pete Pachal: So yeah, go ahead, Patrick.

Patrick Sullivan: Yeah and yeah, maybe I pick that up. I mean you know, I think to to the point there, the these I think we all know tokenomics, you know, the cost of a token is is driving more and more decisions that are being made. And these large LLMs are certainly aware of their token token burn as they are conducting training or retrieval. So to Kim’s point. The the version of your website that looks beautiful to a human is compelling. It has videos, it has a lot of rich content, and it’s bloated from the LLM perspective. A lot of really important information that you want to show up in that LLM on behalf of your customers is sort of below that that token budget, basically, right? So Kim, you know, was alluded you only get about 10,000 tokens. If really, really important information sits below that. It’s invisible. It does not get ingested to the LLM. We’re seeing customers experience massive business impact because they’re not showing up on the LLMs. Their competitors are, you know, they’re seeing, you know, in commerce and travel, you know, huge revenue shifts to competitors. So what what I found when we started down this path was that. you know, there were IT organizations that were building two versions or more versions of their website, one for the human and another one that that was built maybe for you know a bot that likes very efficient HTML. Yeah, another version it’s worse than that. You know, some like Markdown, most like HTML. You know, so so what we do is just sort of on the fly, we’re making sure that that bots that that we know that like HTML

Pete Pachal: A non human.

Kim: And not

Patrick Sullivan: Or getting HTML that’s within their budget. Those that like markdown get markdown. and then there’s also a feedback loop. You know, we talked about that. You detect the bots, categorize, and then you optimize. The optimize for the AI bots is to give them exactly what they want. And then the feedback loop there is, you know, the the metrics that Kim was mentioning. How are you doing relative to competitors for citations, other metrics? And then there could be content, you know, recommendations, you know. it’s not just making sure that everything, as Kim said, everything that that you have published is showing up, but also maybe there’s some things that you haven’t published about that you should, right? So that’s what we see as kind of the full life cycle.

Pete Pachal: Nice. You mentioned on the fly there, I might understand that like if you’re a a site owner of whatever whatever you’re doing, that you can if if I don’t know what you have to enable on your back end, but like the idea that it it’s identifying a bot and then it’s like spinning up kind of a a very efficient version of that page with the relevant information for the bot right then and there. Is that kind of the idea?

Patrick Sullivan: That’s the model. So as that as a human comes in, they get the human version because we detected that that was a human. as, you know, one of these specific AI bots come in, we’re gonna dial up the optimization and serve them, you know, that optimized version that they want and then capture all the analytics that that Kim thrives on and and is pouring over on an hourly basis of you know, you know, how’s the business doing from a a digital marketing perspective? So so that’s all. you know, part of the the platform.

Pete Pachal: So as a publisher, if you’re a publisher and you have content and presumably want to have it in LLMs, or at least the LLMs that you’ve authorized, the I guess are there any tools that are being underutilized in this regard? Do you do it? Like I I’m I I don’t want to get too nerdy here, but it is like I’ve I’ve recently gone to a couple of talks about snippets and that idea of sort of identifying key content.

Patrick Sullivan: Yeah.

Pete Pachal: whether it is on the pa whether it’s lower on the page or not, and just sort of giving emphasise emphasis to certain aspects of the content for the machine. Is that something that you think about as part of the the overall way you implement things? Curious.

Kim: All day, every day we are we are liter literally updating the site every single day for humans and for bots and always thinking about the highest value content that we want to deliver to either audience. And sometimes it’s slightly different, but it it is a it’s not a you know, turn it on and forget it. It’s a it’s an always on ecosystem that we’re fueling and the measurement aspect is critical because you need to understand how those different things you’re serving up are serving you.

Pete Pachal: Sure.

Kim: vis-a-vis your performance and so that constant tuning is key, but having the ability to have the bot site and the human site makes it that much easier. Before it was very manual, very time consuming. And we saw immediate implications against revenue because more people are going to LLMs to buy, you know, whereas SEO is more about research. When you go into those LLMs, you typically know what you want. You’re looking for that answer. And so this gives you the ability to serve it up quickly and correctly to the LLMs.

Pete Pachal: Nice. As we c start to wrap up here, I’d love to ask a couple questions maybe about standards. And I know, you know, there’s a standard called Real Simple Licensing that has been supported by a lot of groups and companies and publishers, I believe including Akamai and Cloudflare. but basically RSL like it allows publishers to essentially like put the machine readable rules for the bots and and search AI training, etc. I’m I’m curious what you think might still need to happen before something like RSL becomes something meaningful and forceable on the web. Patrick, why don’t you take it?

Patrick Sullivan: Yeah, so so I would say in in general, you know, there are there have always been, you know, pages that provide suggestions to bots, you know, this is the part of the website we want you to crawl, you know, here’s what we want you to do elsewhere. you know, some some bots obey those suggestions, other bots kind of treat treat those suggestions like the speed limit sign on the interstate and you know, maybe they follow it, maybe they don’t, but you know, certainly I think it’s it’s definitely best practice to publish those instructions and there are a lot of bots that will follow those. but there are many, many entities out there, many different parties and some some play closer attention to the rules than others.

Pete Pachal: But I guess what happens is even when the sometim sometimes you can’t even agree on the rules, right? I mean, there was the whole row that ver Cloudflare versus Perplexity last year and perplexity was insisting that it would user agents were just behaving like agents and agents can do what they want or not do what they want, but like you know, they were representing people, so they have sort of a different set of rules than search crawlers or training crawlers. And you know, the subtleties of the we don’t have to rehash the whole thing. But I guess that’s the point of these standards. some extent is like sh do we need new rules as we go forward for for managing this stuff. RSL would seem to be something that that like you say it sort of add gives a set of instructions. like you could you could write theoretically a set of instructions for agents, but they might not do it. And is again i i it do we the I guess it’s push coming back to the enforceability problem of like if someone wants them to, w like the what are the tools that or the things that I guess the levers that s someone like Akamai can sort of pull on to sort of help them.

Patrick Sullivan: Yeah, I I I think it’s there are always, you know, bots that that play by the rules and then there are other particularly when it comes to scraping and things like that, there are always bots that we’ve been dealing with for a decade plus that that do not like to play by the rules and you have to detect them and you know, take enforcement into your own hands. But you know, a lot of the the larger organizations y you know, are much more likely to play by the rules.

Pete Pachal: This has been great, guys. Thanks so much for dropping by. For before you go, I’d love to ask you guys a really quick question. I try to ask most of my guests, which is that is there something about this new AI era and agent driven internet that either keeps you up at night or you’re optimistic about? feel free to tell me both. I always like to get optimism and pessimism about you know what’s gonna happen. but why don’t we start with you, Kim, ladies first?

Kim: Sure, I would say what keeps me up at night is it is unprecedented and changing by the minute. I feel like if you take a day off, you’re behind. so so that would be part one. And what I’m most optimistic about is I think, you know, ultimately we’re all striving for a better customer experience, right? And you know, I don’t think there is a better one than LLMs delivering kind of hyper personalization, real time answers. And so I think as quickly as companies can prepare. I think the end game will be better for everybody.

Pete Pachal: Nice. Patrick, what say you?

Patrick Sullivan: Yeah, so so Pete, I think the the pessimism is pretty easy. You you know, just working in security probably multiple hours every day is just working with large organizations to understand the risk of the latest frontier LLMs as it pertains to finding many, many more vulnerabilities that we had seen before. So it’s it’s a pretty rough year to be in in security with just all the vulnerabilities that that everybody’s facing. so Yeah, definitely take care of your applications. They’re having a rough go. on the optimism side, you know, just looking at at sort of the emerging you know, I think you touched on it, frameworks for for agents, you know, for commerce, for other things. and then also, you know, as it pertains to you know, talking to an organization and saying, you know, we don’t have to publish multiple different versions of the website, you know, manually. We can kind of leverage automation there.

Kim: Yeah.

Patrick Sullivan: That’s a fun conversation to have.

Pete Pachal: Nice. Sounds like you might have had early access to mythos. I’m just guessing here. Akamai. You get to get sir get your hands on that, baby?

Patrick Sullivan: you so we’ve been working with the the you know, many of the latest Frontier L LMs, but every security team out there either had access to them or they’re being asked by the board how to contemplate what are the implications, you know, when the the general public, that’s what everybody’s worried about, when the general public gets access, you know, when all these vulnerabilities come home to Roost. So that’s that that’s definitely the one that keeps us up at night, no doubt about it.

Pete Pachal: Nice. Well, if if no one takes if to if you take nothing else from this podcast listeners, set up your two factor off, among other things. guys, this has been great. Thanks so much for spending some time here on the Media Copilot.

Kim: Yeah.

Kim: Thank you.

The post When bots become the audience  appeared first on The Media Copilot.