Cloudflare, Inc. and beehiiv, a newsletter-focused publishing platform, announced a strategic partnership on June 23, which they say will give independent creators greater control over how AI tools access, discover, and use their work.

Cloudflare’s advanced Crawl Control technology is integrated into the beehiiv platform, offering tools to monitor AI crawler activity in real time and manage those crawlers’ access to their content. This integration gives creators two options for managing their digital footprint: publishers can either allow all or some AI search engines to freely crawl their content for greater visibility or completely block AI scraping to protect their archives for future monetization and licensing.

Managing AI bots historically required manually updating robots.txt files or configuring firewalls. The Cloudflare–beehiiv partnership removes these steps, allowing publishers of all sizes, from major media organizations to independent creators, to easily set automated preferences through the platform’s standard dashboard settings. 

Key features of the integration include personalized analytics showing exactly which AI crawlers are accessing their content, which are being blocked, and how much referral traffic they generate, as well as one-click controls to allow or block specific AI models and automatic updates that extend those controls to new crawlers as they appear. 

Tyler Denk, co-founder and CEO of beehiiv, said AI bots from OpenAI, Anthropic, Google, and Microsoft had crawled content hosted on the platform 490 million times. 

“Creators and publishers should own and control their work,” Denk wrote, adding that they “may want to be indexed to maximize AI discovery” or “may want to keep [their] valuable content private and not crawled by AI bots.” 

Matthew Prince, co-founder and CEO of Cloudflare, echoed Denk’s statement: “As the internet evolves, [Cloudflare’s] commitment remains the same: ensuring creators have the tools they need to thrive,” Price said. “This partnership with beehiiv is the next logical step in that mission, giving newsletter operators the transparency and control to navigate the AI era on their own terms.”

AI Crawl Control has now launched in beta for all beehiiv users, giving publishers new visibility into  how AI systems access their content and the traffic those crawlers generate.  

The post Creators get new say over AI scraping through Cloudflare–beehiiv partnership  appeared first on The Media Copilot.

Reuters and Time are blocking all AI bots by default and only letting approved crawlers through—a whitelist approach that more publishers are adopting as the volume of unauthorized scraping grows.

As Digiday reports, both publishers moved to block AI bots last month, joining People Inc. and The Atlantic, which adopted similar strategies earlier this year and late last year respectively. The goal is simple: content costs money to produce, and AI companies have been taking it without paying.

“We saw that there was an imbalance between the value that publishers like Reuters provide and the value that Reuters receives in kind, and so instead we went from a default allow-all to a default disallow all,” said Josh London, head of Reuters Professional, which oversees the direct-to-consumer and direct-to-professional businesses. Reuters has since signed AI licensing agreements with Microsoft and Meta, according to the report.

The publishers aren’t relying on any single tool. Reuters uses robots.txt files, a method that is voluntary and non-binding, and one that many AI bots simply ignore. The approach is meant to create friction and signal that access requires negotiation. “If you want this, let’s have a conversation and then we can allow you to access,” said Alphonse Hardel, head of agency at Reuters, who leads the content licensing business.

Time allows roughly 70 bots on its site, ranging from AI lab crawlers and social platforms to its own operational systems. The volume of bot traffic has become significant enough that Time sees it as leverage for a future AI visibility product it’s developing for brand clients.

The economics are also shifting. Blocking bots cuts server costs: Hardel said the expense of the bot-blocking vendor can be nearly offset by the reduction in non-human traffic. At People Inc., the shift from a block list to an allow list meant going from blocking roughly 2,100 user agents to over 30,000, said Lindsay Van Kirk, the company’s SVP of innovation, speaking at an IAB Tech Lab event in May.

“Adding two full seconds of latency to the majority of scrapers when you implement a block-all-bots approach is a really good thing, even if they have to go through,” Van Kirk said. “Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business.”

The IAB Tech Lab has published guidance on bot management, and the SPUR Coalition—a publisher group formed earlier this year with major news organizations—announced significant new membership as it works to create technical standards for AI licensing and content protection.

For Reuters, the change hasn’t reduced site traffic. After monitoring bot activity over an extended period, the company had enough data to identify which bots it could block without hurting revenue. The publisher maintains a public robots.txt file that lists approved bots, a benchmark that also supports enforcement discussions, said Phil Andraos, general manager of Reuters Digital.

“It’s not a set it and forget it approach,” London said. “The value of content is something that we ignore at our own peril, especially as AI scales.”

The post Reuters and Time flip the script on AI bots with blocking whitelists appeared first on The Media Copilot.

The copyright fight between publishers and AI companies has many fronts, but the trickiest one comes down to a single word: outputs. Even if scraping feels indefensible, courts generally aren’t interested in punishing the scrapers unless the resulting product is doing measurable damage to the people whose work was taken. Civil claims especially need a clear line from the act to the injury.

The 2023 Sarah Silverman case is the textbook example. A group of authors including the comedian sued OpenAI for using their books without permission, and a judge later tossed several of the claims because the plaintiffs couldn’t point to specific outputs that were direct copies of their work. Knowing a large language model (LLM) ingested your writing isn’t enough on its own. You have to show the model is producing something that eats into your business.

Why outputs matter more than scraping in court

That evidentiary burden is part of why these cases struggle. Scraping happens silently, at machine speed, behind layers of infrastructure most publishers never see. The outputs of public-facing tools like ChatGPT, Gemini, and Perplexity are easy enough to inspect, but a much larger scraping economy operates outside that view.

It’s been an open secret for a while that AI companies pull data from third-party brokers, and media analyst Matthew Scott Goldstein recently put numbers to it. His report, covered in Digiday, identifies at least 21 companies, several backed by hundreds of millions of dollars, that routinely scrape publisher content without paying for it and sell their “data services” to customers that include OpenAI, Amazon, and even publishers like The Telegraph.

The report is essentially a map of what scraping looks like when no one stops it. Multimillion-dollar businesses, most of them obscure to readers, exist for the sole purpose of indexing publisher content and reselling it to bots and agents. The names won’t ring bells: Parallel AI, Exa, and Bright Data. And they aren’t hiding what they do. A recent Wall Street Journal profile describes Parallel AI as a platform “dedicated to servicing AI agents.” Goldstein calls it a “scraper company with better branding.”

Charlie Munger’s old line—show me the incentives, and I’ll show you the outcome—applies cleanly here. Between the losing streak in court and an administration that has openly waved off copyright concerns, the signal to AI companies and the brokers feeding them is unmistakable. Unauthorized scraping carries little risk, and the default settings of the system push toward more access, not less.

The bot-blocking decision every publisher faces

That setup leaves publishers between a rock and a hard place. Either you block bots as aggressively as your stack will allow, or you let them in. Letting them in feels like surrender, but it also ends the constant whack-a-mole and clears space to build a business that assumes AI will ingest and repurpose your work no matter what.

I’d argue those two stances aren’t as opposed as they look. Publishers should defend their copyright, but they also have to plan for a world in which AI engines are baked into how content reaches anyone. AI is now a distribution channel, a middle layer, and an audience all at once.

So what does a serious response to all this look like? Five components, in my view. Not every publisher will have the resources for all of them.

• Get better at blocking bots. IP protection takes both legal and technical effort. Most large publishers are nominally blocking bots, but doing it for real means going past the robots exclusion protocol, the polite instructions sites give bots and which bots regularly ignore. People Inc. CEO Neil Vogel has said his company has needed to become highly sophisticated at blocking unauthorized bots.
  
  Smaller publishers won’t have that level of resourcing, but technical partners exist, and infrastructure providers like Cloudflare have started shipping copyright-protecting defaults. Even when sophisticated blocking is out of reach, intel is not. Look at your bot traffic, but also audit the AI services themselves to see where your content has surfaced without permission.
• Practice good GEO. This one feels backwards at first. Whether or not bots have your permission, your content should still be readable to them. Access is binary, on or off. Ignoring generative engine optimization (GEO) just means your work is harder for every bot to parse, including the ones you’d want to let in.
  
  The case for GEO is practical. Scraping is happening, so you may as well compete inside the summaries and pick up whatever qualified traffic results. It also generates a paper trail for the audits in the previous bullet, which can support any future legal claim. And it becomes foundational if you ever build an in-house agent or MCP server on top of your content.
• Shift your business model. I’ve covered this at length before, so the short version. The Google-era model is shrinking, and any business built on monetizing anonymous traffic is shrinking with it. New revenue streams (events, subscriptions, data products, licensing) have to be cultivated. Easier said than done. Diversification has to become a religion for ad-dependent publishers, not a side project.

• Sue. Not realistic for every publisher. Going after OpenAI or Perplexity requires resources most newsrooms don’t have. But the Goldstein report effectively introduces a new set of potential defendants who have been mostly invisible until now. Given what they’re openly doing and the size of the market involved, it would be strange if more legal action didn’t follow.

• Lobby for regulation. Federal action looks unlikely in the current climate, but states are moving on AI policy, including transparency and disclosure rules around training data. Real progress may not require rewriting copyright law from scratch. Even something as simple as requiring bots to properly identify themselves would stop the impersonation that makes the current scraping economy possible.

Why agency matters more than victory

As bots keep “eating the internet,” it’s tempting to treat scraping as one more thing publishers just have to live with. Some of that resignation is earned. But inevitability is not the same as paralysis. In a world increasingly run by agents, publishers have to claim back some agency of their own. Protect what’s protectable, adapt where adaptation is the only path, and refuse to let the same companies that scraped your work also write the rules for what happens to it next.

A version of this column appears in Fast Company.

The post Inside the AI scraping economy nobody wants to talk about appeared first on The Media Copilot.

ChatGPT made its debut when I was Chief of Staff at CoinDesk, and I ended up leading the publication’s approach to AI. Our internal AI committee quickly ran into an argument that felt simple on the surface but was surprisingly thorny underneath: Should we let AI systems index our articles, or shut the door?

The early consensus leaned toward “block them.” Even before the current wave of copyright fury, the logic was obvious: why hand over original reporting to AI companies that could summarize it, monetize the attention, and send us nothing but vibes in return? If compensation was the only language anyone understood, then withholding content felt like leverage.

But one person made the counterargument: If AI becomes the default way people retrieve information, he said, opting out doesn’t protect your work—it removes it from the record that AI users will actually see. Your reporting still exists, but it’s absent from the synthesized “answer” that becomes the starting point for everyone else. And if your stories aren’t in that amalgam, your competitors’ framing will be. The loss wouldn’t just be referrals (which, even then, we assumed would be minimal). It would be authority.

The cost of silence

At the time, the debate felt like a forward-looking thought experiment. Nearly three years later, it’s become the centerpiece of the media’s AI conversation. Presence inside AI summaries—whether you’re a publisher, a brand, or a PR team—is suddenly both urgent and poorly understood. Publishers are still fighting over copyright and compensation, and they should. But whatever the courts decide, a large and growing audience now encounters journalism through an interpreter. There’s a reason traffic to publisher sites dropped by a third in 2025: To many, AI is the new front door of the internet.

Back then, we didn’t have a label for the “let them in” strategy. Now we do: generative engine optimization or GEO (and yes, sometimes the first word gets swapped for “answer,” or AEO). When I’ve written about GEO before, it’s been framed as a pragmatic question: why participate in a system that summarizes your work and keeps people from clicking through?

But that question is backwards. The more useful one is harsher: what’s the price of refusing? And the answer is influence—specifically, your influence over what becomes consensus inside your domain.

The risk isn’t the loss in traffic—that will ebb away no matter what. Audiences are turning to AI as their information guides no matter what publishers do. What’s on the line is who gets to be the interpreter when the audience shows up somewhere else. For decades, journalism has set the baseline by reporting facts, validating claims, and establishing what’s known. That doesn’t just inform readers; it gives everyone else something solid to respond to. If those inputs aren’t present in the AI layer, the machine will still produce a picture of reality—just not a very good one.

And here’s the uncomfortable part: even if you opt out of AI summarization, your information won’t. Someone will re-report it. Someone will rewrite it. Someone will publish their own version, and some of them won’t block crawlers.

A foundational concept of copyright law is that, although works are copyrightable, the underlying facts and ideas aren’t. Except now those facts travel through someone else’s lens first. That lens becomes the “first draft” the machines reuse. Will it be incomplete? Probably. Will it still harden into the default answer as AI use expands? Also probably.

This is why the “block or die” framing misses what’s actually happening. Blocking AI from indexing your content means blocking yourself from having a say in what a rapidly expanding portion of the world counts as truth. GEO, in that sense, isn’t a growth hack. It’s a recognition that the old scoreboard—traffic, time on site, even subscriptions as a direct outcome of a single session—doesn’t capture the new fight. The battleground is the summary. The trophy is citation, narrative presence, and the long-tail compounding of trust.

Shaping truth at scale

None of that is an argument for surrender. Publishers shouldn’t shrug and let AI companies crawl anything they want for free. If anything, proving that your reporting is driving the consensus inside AI answers is the most concrete evidence of value you could ask for. The legal fights tend to focus on consent, copyright, and compensation. Fair enough. But GEO makes the deeper contest obvious: Who gets to shape meaning at scale?

Right now, it’s difficult to show—cleanly, repeatably—how specific pieces of content echo through AI-generated answers. That’s changing fast. Marketers, PR agencies, and brands are already pushing to measure GEO, refine it, and turn it into a playbook that blends content strategy, technical signals, and deliberate communication. Like SEO, it will always be more art than science. But by this time next year, I’d bet GEO won’t feel nearly so embryonic.

And AI itself will be an even bigger informational gatekeeper than it is today. Compensation matters, and the media shouldn’t stop demanding it. But litigation can’t be the only plan. Publishers also have to compete to be present inside the new crucible where truth is forged—because that crucible isn’t going away.

Journalists may no longer control the interfaces where people get information. But they still control the facts. The job now is to make sure those facts survive translation—and that the story that gets told at scale still has your fingerprints on it. Fighting for a better system doesn’t mean opting out of the one people are already using.

A version of the column first appeared in Fast Company.

The post AI isn’t taking your clicks. It’s taking your credibility appeared first on The Media Copilot.

Publishers implementing bot monitoring tools face a data paradox. TollBit helps quantify AI scraping by analyzing traffic patterns, visitor identification and access logs—the same information that raises privacy concerns when processed by third-party platforms. Understanding which bots harvest content requires tracking who accesses what, when and how often.

Digital Trends implemented TollBit’s monitoring without major security concerns. The platform operates similarly to Google Analytics—tracking visitor behavior through lightweight JavaScript tags without accessing backend systems. But publishers considering adoption should understand what data gets processed, how TollBit handles that information and what risks remain even with standard security controls.

Risks identified in TollBit’s data processing

The primary risk with any analytics platform involves unintended data exposure through inadequate security controls, unauthorized access or service provider breaches. TollBit processes visitor IP addresses to distinguish bots from humans, access logs revealing which pages get scraped and traffic patterns showing scraping frequency over time.

For most publishers, this data processing parallels existing analytics tools. Google Analytics, Adobe Analytics and similar platforms already track visitor IPs, pageview patterns and referral sources. TollBit adds bot-specific monitoring without expanding the fundamental data collection publishers already conduct.

However, the licensing features introduce additional considerations. When publishers activate bot paywalls, TollBit handles transaction processing—metering content access, processing payments and managing invoicing. This financial layer adds payment data and commercial relationships to the information TollBit processes on publishers’ behalf.

Documentation doesn’t specify data retention periods beyond standard processing needs. Publishers with formal data destruction policies—mandated timelines for purging visitor logs, regulatory requirements around analytics data—need clarity on exactly how long TollBit retains IP addresses, access patterns and transaction records.

The bot detection methodology itself creates potential exposure. Identifying scrapers requires analyzing traffic patterns that might inadvertently capture information about human visitors misclassified as bots or legitimate tools flagged incorrectly. Misconfiguration could block accessibility services, research tools or other authorized access that publishers want to permit.

Security controls TollBit has implemented

TollBit operates as a data processor under a Data Processing Agreement with publishers. The platform processes limited personal data—primarily visitor IPs for bot detection—under publisher instructions rather than for independent purposes. The company states it doesn’t sell or share that personal data and uses subprocessors subject to security and contractual controls.

The monitoring implementation uses JavaScript tags similar to Google Analytics, operating at the application layer without requiring backend system access. This architecture limits exposure to frontend analytics data rather than sensitive backend systems, databases or user accounts.

For Digital Trends’ implementation, security considerations proved minimal. The monitoring tracks publicly visible traffic patterns—which pages get accessed, how frequently, by which identifiable bots. No confidential editorial content, unpublished materials or sensitive business data flows through TollBit’s systems.

Publishers activating monetization features should review TollBit’s Publisher Terms of Service for complete data processing details. The transaction infrastructure introduces payment processing—a regulated activity with specific security and compliance requirements beyond basic analytics.

The platform’s security posture reflects standard analytics practices rather than specialized protections for sensitive materials. Publishers comfortable with Google Analytics’ data handling will find TollBit’s approach comparable. Organizations with stricter requirements than standard analytics tools provide need custom data processing agreements or on-premises alternatives.

Security checklist for TollBit users

Before implementing TollBit’s monitoring or licensing features, verify the following:

• Does your organization’s privacy policy permit third-party traffic analytics processing visitor IPs?
• Are you comfortable with data processing equivalent to Google Analytics (JavaScript tags, visitor tracking, access logging)?
• Do you have formal data retention policies requiring specific purge timelines for visitor logs?
• Would bot misclassification accidentally blocking legitimate accessibility tools or research access violate your editorial principles?
• If activating monetization, does your organization require specific payment processing compliance (PCI-DSS, financial regulations)?
• Do you need custom Data Processing Agreements specifying retention, deletion and breach notification beyond standard terms?
• Are you subject to regional data protection regulations (GDPR, CCPA) requiring specific visitor consent for analytics tracking?

Organizations answering “yes” to formal retention policies, payment compliance requirements or regional data protection regulations should review TollBit’s Publisher Terms of Service and potentially request custom Data Processing Agreements before implementation.

Publishers handling public-facing content without unusual security requirements will find TollBit’s monitoring comparable to existing analytics tools. The platform adds bot-specific visibility without fundamentally changing data processing practices most publishers already conduct.

Organizations can review TollBit’s complete data processing and privacy terms at tollbit.com. For most publishers implementing monitoring only, security considerations parallel standard analytics tools without introducing novel risks.

Frequently Asked Questions

The post What publishers need to know about TollBit’s data handling appeared first on The Media Copilot.

Publishers face declining search traffic as AI overviews replace direct links. Bots scrape content at scale without compensation. Traditional business models—display ads, affiliate links, subscription paywalls—don’t address autonomous agents harvesting articles without delivering referrals.

TollBit and ProRata both target this revenue gap, but through fundamentally different mechanisms. TollBit monetizes bot access by creating a licensing infrastructure in which AI companies pay to scrape content. ProRata monetizes on-site usage by sharing ad revenue generated from AI answers that cite publisher content.

The question for publishers: Which model generates income faster?

TollBit’s licensing infrastructure

TollBit operates as a marketplace for bot access. Publishers set prices per 1,000 pages scraped, creating paywalls that require AI companies to pay before consuming content. The platform offers two license types: summarization use (for citations and grounding) and full display (complete article text). Neither permits model training.

Implementation takes under 30 minutes using JavaScript tags and DNS configuration. Digital Trends completed setup quickly and now monitors 4.1 million weekly scrapes, with ChatGPT accounting for 87.8 percent of bot traffic. The free monitoring reveals a 966-to-1 extraction ratio—bots taking content without delivering referrals.

But Digital Trends generates zero revenue from TollBit. Monitoring provides value, but monetization requires activating paywalls and—critically—AI companies willing to pay. That marketplace hasn’t materialized at scale.

The model aligns with existing intellectual property frameworks. Publishers already license content through syndication and republishing agreements. Bot licensing extends familiar practices. Local news outlets publishing unique, irreplaceable content—school closures, municipal meetings, hyperlocal coverage—could command premium pricing for information available nowhere else, according to TollBit co-founder Olivia Joslin.

ProRata’s attribution and ad-sharing model

ProRata avoids the chicken-and-egg problem TollBit faces by generating revenue from ads served alongside AI answers rather than from AI companies licensing access. Publishers implement on-site AI search tools (such as Gist Answers) that generate AI responses using licensed content. Ad revenue gets split 50/50 between ProRata and publishers, with publisher shares allocated based on each source’s contribution to responses.

This model doesn’t require blocking bot access or enforcing paywalls. Publishers can implement ProRata alongside traditional SEO strategies, open-access models, or existing paywalls. The on-site AI search complements rather than restricts external bot traffic.

Integration provides attribution reporting showing where publisher content appears in AI answers, visibility into which articles contribute most to responses, and on-site AI search tuned to specific content. These features deliver utility independent of revenue generation.

But actual revenue depends on audiences using the on-site search tool and ad rates for AI-generated content—metrics ProRata hasn’t disclosed publicly.

Core operational differences

The platforms capture value at different points. TollBit charges AI companies for scraping content. ProRata shares ad revenue from AI answers generated for human visitors. This difference determines implementation complexity and the timing of revenue.

TollBit requires bot access policies, allowlist maintenance and licensing terms before monetization activates. Revenue depends on industry-wide marketplace maturation—multiple publishers and AI companies participating in paid licensing. Publishers control monitoring, but don’t control when income materializes.

ProRata requires integrating on-site AI search and implementing ad systems. Revenue depends on individual site implementation and audience adoption—factors publishers control more directly. Income is generated when visitors use the search tool, not when industry licensing markets mature.

Neither platform has disclosed revenue data at scale. TollBit’s monitoring-only implementations generate zero income. ProRata’s 50/50 split sounds attractive, but actual revenue depends on on-site search traffic volume—figures the company hasn’t released.

Which model suits your strategy

TollBit suits publishers willing to implement infrastructure now for speculative revenue later. The free monitoring provides immediate value by providing insights into bot behavior, extraction patterns, and traffic sources. This requires patience and tolerance for uncertain timing.

Digital Trends exemplifies this approach: monitoring reveals extraction patterns informing editorial strategy while licensing infrastructure waits for marketplace development.

ProRata suits publishers wanting immediate revenue. The on-site AI search needs users, but ad revenue doesn’t depend on AI companies licensing content—a potentially faster path to income.

Neither platform guarantees revenue. Publishers should evaluate both models against traffic patterns, content uniqueness and tolerance for speculative positioning.

Frequently Asked Questions

The post Two paths to AI revenue: Licensing bot access versus sharing ad income appeared first on The Media Copilot.

Dan Gaul helped launch Digital Trends in 2006 during what he calls the “Wild West days” of online publishing. The formula was simple: create content, optimize for Google, watch the traffic roll in. Display ads and affiliate links funded the operation. The clicks kept coming.

That model is collapsing. When Gaul pulls up TollBit’s analytics dashboard today, the numbers tell a brutal story. In the past week alone, Digital Trends received 4.1 million bot scrapes. During that same period, AI chatbots referred just under 4,200 human visitors back to the site.

“966 AI scrapes to one referral,” Gaul says. “It’s crazy.”

Digital Trends, based in Portland, covers consumer electronics, smart home technology, gaming and lifestyle content through its flagship site and eight additional verticals. The company remains privately held and independent nearly two decades after founding. But the shift from search traffic to AI overviews has been devastating. The company once employed 170 people, most in editorial roles. Today it relies primarily on freelancers.

“It’s really hard to maintain a huge workforce when we don’t have private equity or large media dollars behind us,” Gaul says.

The fundamental equation changed. AI applications read dozens of articles per query but deliver minimal referral traffic. Bot traffic increasingly outnumbers human pageviews. Some hosting providers now charge publishers for bandwidth consumed by scrapers—a double punishment where publishers lose human traffic while paying to serve bots. TollBit offered a way to quantify the problem and potentially monetize what was otherwise pure extraction. Implementation took under 30 minutes and cost nothing upfront.

Implementing lightweight monitoring without backend access

Gaul started with TollBit’s monitoring features, which require only a JavaScript tag similar to Google Analytics. The setup demands no backend system access or complex technical integration. For Digital Trends, implementation took under 30 minutes—mostly DNS configuration that any IT team could handle.

TollBit operates on a “three Ms” framework: Monitor, Manage, Monetize. Co-founder Toshit Panigrahi designed the platform to address how the web’s primary consumers shifted from humans to autonomous agents. Traditional analytics tools like Google Analytics track human behavior but miss bot activity entirely. Publishers couldn’t quantify what was happening to their traffic.

The monitoring dashboard reveals which AI services and bots access content, how frequently, which specific pages get scraped and how many human referrals arrive in return. This data doesn’t exist in standard analytics. Google Analytics counts human sessions. TollBit counts the bots reading content without sending humans back.

For Digital Trends, the analytics immediately quantified what Gaul suspected. ChatGPT’s crawler accounted for 3.6 million scrapes over one month—87.8 percent of all AI bot traffic to the site. The homepage got scraped most frequently as bots checked for fresh content. Individual articles showed surprising patterns. The top scraped article in October 2025 was “Instagram finally fixes the one thing you hated about Reels”—a timely news story without lasting SEO value.

Gaul remains unsure why that particular article spiked bot traffic, but he’s hoping monitoring will reveal long-term trends over time.

Understanding which content AI extracts most aggressively

The analytics revealed patterns about which content types AI scrapers target. Evergreen content like “how to take a screenshot on Windows” took the hardest traffic hit because AI can answer those queries with specific step-by-step instructions without attribution. Reviews of consumer electronics similarly suffered as AI aggregates specs and ratings from multiple sources.

“Information that is data-centric or spec-centric I think is where publishers are getting hurt most,” Gaul says. “And where they’re getting hurt the least is more on the featured, editorial, creative writing stuff.”

This distinction matters for editorial strategy. Content easily summarized or spec-heavy faces maximum AI competition. Original analysis, narrative features and creative work face less substitution because AI can’t replicate unique perspectives as easily as it aggregates facts.

Bandwidth costs compound the problem. Some hosting providers charge publishers for traffic consumed by bot scrapes. Digital Trends avoided this through their provider choice, but Gaul heard from other publishers facing the double bind: losing human traffic while paying to serve automated scrapers.

TollBit’s monitoring helps publishers track CDN costs from automated traffic and identify the specific bots driving bandwidth consumption. This data informs both technical decisions—which providers penalize bot traffic—and business decisions about whether enforcement or licensing makes more sense than absorption.

Evaluating monetization without implementing paywalls yet

Digital Trends hasn’t activated TollBit’s bot paywall or monetization features. The platform offers two licensing types publishers can price independently: summarization licenses allowing AI products to use content once for citations or grounding, and full display licenses closest to lightweight syndication rights. Neither permits model training.

Publishers set rates per 1,000 pages accessed. TollBit handles transaction infrastructure—metering, checkout, invoicing—while charging AI companies a small fee on top of publisher-set prices. The revenue-sharing model means publishers pay nothing upfront or monthly. Costs come only from the AI companies accessing content.

Gaul remains realistic about monetization potential. “We’re just not there yet in terms of the whole AI ecosystem,” he says. The licensing marketplace remains nascent. Revenue is unpredictable. But the infrastructure matters for future positioning as the market matures.

Panigrahi argues uniqueness drives premium pricing: “If you have an article that is irreplaceable, that you cannot find anywhere else, you can command a very high premium.” This applies particularly to local news outlets publishing information—school closures, municipal meetings, community events—that appears nowhere else on the web.

The team at TollBit thinks bot traffic will only increase. “These AI companies are coming tens of thousands of times a day to your site. It’s not that they just crawled it once, took the content and left. They need to access this regularly,” Panigrahi says.

Balancing enforcement with legitimate search access

Implementation requires policy decisions beyond technical setup. Publishers must define which bots get allowlisted—approved for free access—versus which face paywalls. Search engines like Google need free access for SEO to function. Legitimate research tools, accessibility services and archival projects may warrant exceptions.

Misconfiguration risks accidentally blocking tools publishers want accessing content. TollBit handles technical enforcement through subdomain routing and robots.txt rules, but publishers determine policy: default action for unauthorized bots (block entirely or redirect to paywall), allowlist maintenance and published terms.

These decisions involve legal and finance teams, not just editorial or technical staff. TollBit’s documentation recommends inviting stakeholders early to review terms and configure payout methods—even if monetization activation comes later.

The platform requires ongoing vigilance. Bot spoofing remains a challenge as unauthorized scrapers masquerade as legitimate browsers. TollBit provides detection tools, but publishers must monitor dashboards regularly to catch evolving evasion tactics.

What didn’t work—and realistic expectations

Digital Trends’ experience highlights limitations in current AI licensing economics. The platform provides sophisticated monitoring and enforcement infrastructure, but the marketplace connecting publishers to AI companies willing to pay remains underdeveloped.

Documentation doesn’t specify implementation challenges Digital Trends encountered beyond the broader reality that monetization hasn’t materialized at scale. The “knowledge value” Gaul emphasizes—understanding bot behavior patterns—represents the current payoff rather than direct revenue.

Publishers considering TollBit should calibrate expectations accordingly. The monitoring delivers immediate value. The licensing infrastructure positions publishers for future opportunities. But guaranteed revenue justifying implementation time remains speculative.

The results

Digital Trends’ TollBit implementation provided detailed analytics quantifying AI scraping patterns previously invisible in standard analytics tools:

• Bot scrapes tracked: 4.1 million in one week
• Human referrals received: 4,200 in the same period
• Scrape-to-referral ratio: 966 to 1
• ChatGPT dominance: 87.8% of all AI bot traffic (3.6 million monthly scrapes)
• Revenue generated: None—monitoring only, no monetization activated yet

Gaul’s assessment remains measured: “The value of TollBit is the knowledge.” The data informs strategy without delivering income.

What’s next for Digital Trends

Gaul’s editorial strategy has shifted from traffic optimization to community building. “At the end of the day, it’s going to be about community. How do you build community without relying on Google?”

The monitoring data informs this pivot by revealing which content types face maximum AI substitution versus which retain unique value. Featured editorial and creative writing suffer less extraction than spec-heavy product reviews—a signal about where to invest scarce editorial resources.

(Inference based on documentation: Digital Trends’ monitoring-only implementation suggests future monetization activation depends on AI licensing marketplace maturation. As more publishers join TollBit’s network and collective bargaining power increases, implementing bot paywalls and pricing to capture compensation for millions of weekly scrapes becomes more viable. The current approach—gather data, understand patterns, position for future revenue—reflects realistic assessment of marketplace timing rather than technical limitations.)

Publishers can explore TollBit’s free monitoring and licensing infrastructure at tollbit.com. Setup takes under 30 minutes with no upfront costs. The platform works best for outlets willing to invest monitoring time now for knowledge value and potential revenue later.

The post TollBit can monitor AI bot scraping, track referral traffic declines appeared first on The Media Copilot.

Dan Gaul, the co-founder and chief technical officer of Digital Trends, suspected AI was gutting his site’s search traffic. Traditional analytics showed referrals dropping as AI overviews replaced direct links. But Google Analytics tracks human visitors, not the bots harvesting content at scale. Gaul couldn’t quantify the extraction.

TollBit’s free monitoring revealed the brutal math: 4.1 million bot scrapes in one week delivered only 4,200 human referrals. ChatGPT alone accounted for 87.8 percent of automated traffic. The data doesn’t generate revenue yet, but it quantifies what was invisible.

“The value of TollBit is the knowledge,” Gaul says.

The gist

Digital Trends’ lightweight implementation exposed the AI extraction economy:

• Implementation required under 30 minutes with no upfront costs
• Analytics revealed 966-to-1 scrape-to-referral ratio documenting traffic asymmetry
• Monitoring identified content types facing maximum AI substitution

How they did it

Digital Trends’ implementation prioritized understanding bot patterns before enforcement:

• Installed JavaScript tracking tag: Added lightweight monitoring code similar to Google Analytics, requiring no backend access or complex technical integration.
• Configured DNS settings: Completed minimal DNS configuration enabling TollBit to route detected bots appropriately—implementation handled by any IT team in under 30 minutes.
• Established baseline metrics: Monitored which AI services accessed content, scraping frequency, specific pages targeted and human referral ratios revealing value exchange.
• Analyzed content patterns: Used data to identify which article types faced maximum extraction—evergreen how-tos and spec-heavy reviews versus original analysis and narrative features.
• Informed editorial strategy: Applied monitoring insights to content planning, recognizing spec-centric information faces AI substitution while creative writing retains unique value.

Key numbers

• Implementation time: Under 30 minutes for monitoring setup
• Cost: Free for publishers (no upfront fees or monthly subscription)
• Bot scrapes tracked: 4.1 million in one week
• Human referrals received: 4,200 in same period
• Scrape-to-referral ratio: 976 to 1
• ChatGPT dominance: 87.8 percent of all AI bot traffic (3.6 million monthly scrapes)
• Revenue generated: None—monitoring only, monetization not activated

What to watch for

Implementation considerations and realistic expectations:

• No guaranteed revenue: Licensing marketplace remains nascent—monetization is speculative positioning for future opportunities, not immediate income stream.
• Requires ongoing monitoring: Dashboard review and policy development demand attention—free implementation doesn’t mean zero time investment.
• Misconfiguration risks: Accidentally blocking legitimate search engines or accessibility tools could harm SEO—allowlist maintenance requires vigilance.
• Bot spoofing continues: Unauthorized scrapers masquerade as browsers—detection requires ongoing adaptation as evasion tactics evolve.

Publishers can explore TollBit’s free monitoring at tollbit.com. The platform delivers value through visibility into bot behavior, with potential revenue dependent on AI licensing marketplace maturation.

The post AI Web Scraping: The Invisible Threat to Websites appeared first on The Media Copilot.

Publishers watching traffic collapse face a measurement gap. Google Analytics tracks human visitors but misses the autonomous bots harvesting content at scale. AI applications read dozens of articles per query without delivering proportional referral traffic. Some publishers suspect the problem but can’t quantify it. Others remain unaware bot scrapes vastly outnumber human pageviews.

TollBit addresses this visibility gap through specialized analytics revealing bot behavior patterns traditional tools miss. Dan Gaul’s experience at Digital Trends illustrates the value: 4.1 million bot scrapes in one week delivered only 4,200 human referrals—a 966-to-1 extraction ratio impossible to measure without bot-specific monitoring.

Three elements drive TollBit adoption for publishers seeking to understand and potentially monetize AI traffic: zero upfront cost removing budget barriers, monitoring capabilities that quantify previously invisible bot activity, and licensing infrastructure positioning publishers for revenue as the AI marketplace matures.

1. Free implementation removes the primary barrier to experimentation

Analytics platforms typically require monthly subscriptions publishers must justify against competing budget priorities. Specialized monitoring tools for bot traffic aren’t free. Implementation costs—technical integration, staff training, ongoing maintenance—compound the direct expense. Cash-strapped outlets can’t justify speculative spending on tools that might reveal problems without providing solutions.

TollBit eliminates this barrier through free access for publishers. The monitoring features require no upfront payment or monthly fees. Implementation takes under 30 minutes—mostly lightweight DNS configuration any IT team can handle. The platform charges only when publishers activate monetization, taking a small transaction fee from AI companies paying to access content.

This revenue-sharing model means publishers pay nothing to experiment. Digital Trends implemented monitoring to quantify bot traffic without budget approval processes or recurring cost justification. For outlets investigating whether AI scraping affects their traffic, the financial risk approaches zero.

The free tier doesn’t compromise monitoring capabilities. Publishers get full analytics revealing which bots access content, scraping frequency, specific pages targeted and human referral ratios. The data exists nowhere else—Google Analytics and similar tools track human behavior exclusively.

For Digital Trends, this revealed ChatGPT accounting for 87.8 percent of all AI bot traffic with 3.6 million monthly scrapes. Understanding that concentration informs both technical decisions—which bots to prioritize for enforcement or licensing—and business strategy about future monetization potential.

2. Specialized analytics quantify the invisible extraction economy

Traditional analytics tools weren’t designed for autonomous agents. Google Analytics, Adobe Analytics and similar platforms track human sessions, pageviews and referrals. Bot traffic gets filtered out or misclassified. Publishers can’t distinguish between legitimate crawlers, malicious scrapers and AI applications harvesting content at scale.

TollBit’s monitoring addresses this specifically. The dashboard reveals bot traffic separately from human visitors, tracks which AI services access content most aggressively, identifies most-scraped pages and calculates referral ratios showing the value exchange—or lack thereof.

For Digital Trends, the analytics immediately quantified what Gaul suspected but couldn’t prove. The 966-to-1 scrape-to-referral ratio documents the extraction dynamic: AI reads nearly a thousand articles for every human visitor it sends back. This data informs editorial strategy by revealing which content types face maximum substitution.

“Information that is data-centric or spec-centric I think is where publishers are getting hurt most,” Gaul says. “And where they’re getting hurt the least is more on the featured, editorial, creative writing stuff.”

Evergreen content like “how to take a screenshot on Windows” gets harvested because AI can answer those queries without attribution. Consumer electronics reviews suffer as AI aggregates specs from multiple sources. But original analysis and narrative features face less substitution because AI can’t replicate unique perspectives as easily as it compiles facts.

The monitoring also tracks bandwidth costs from bot traffic. Some hosting providers charge publishers for scraper-consumed bandwidth—a double punishment where publishers lose human traffic while paying to serve bots. TollBit helps identify which specific bots drive CDN costs, informing decisions about whether enforcement, licensing or provider changes make most sense.

3. Licensing infrastructure positions publishers for emerging revenue streams

The AI licensing marketplace remains nascent. Revenue from bot paywalls is speculative. But infrastructure matters. Publishers implementing licensing systems now position themselves for compensation as the market matures and AI companies face increasing pressure to pay for content they extract.

TollBit provides transaction infrastructure publishers couldn’t build independently. The platform handles metering, checkout, invoicing and marketplace discovery where AI companies can license content from multiple publishers without individual negotiations. Publishers set prices for two license types: summarization licenses for citations and grounding, and full display licenses for complete article text. Neither permits model training.

This marketplace model mirrors how Spotify created infrastructure for music licensing after Napster demonstrated demand for digital distribution. TollBit co-founder Toshit Panigrahi frames the platform as addressing a similar transition: “If we are in the Napster days, what does the Spotify solution look like? If someone wanted to behave well, how do they come and get access to those articles?”

Digital Trends hasn’t activated monetization yet. Gaul remains realistic: “We’re just not there yet in terms of the whole AI ecosystem.” But positioning matters. As bot traffic increases—Panigrahi notes AI companies access sites “tens of thousands of times a day”—publishers with licensing infrastructure ready can activate revenue streams when marketplace conditions improve.

Panigrahi argues uniqueness drives premium pricing: “If you have an article that is irreplaceable, that you cannot find anywhere else, you can command a very high premium.” This applies particularly to local newsrooms publishing information that appears nowhere else—school closures, municipal meetings, hyperlocal coverage AI can’t aggregate from multiple sources.

Who should consider TollBit

Publishers experiencing search traffic decline without understanding the role of AI overviews gain immediate value from monitoring. The analytics quantify what traditional tools miss—revealing whether bot scrapes correlate with referral drops and which content types face maximum extraction.

Outlets willing to invest monitoring time now for potential revenue later represent TollBit’s target audience. The free implementation means minimal risk, but dashboard monitoring and policy development require ongoing attention. Publishers needing guaranteed immediate ROI to justify any new platform won’t find that here.

Organizations experiencing rising CDN costs from automated traffic benefit from data identifying which bots drive bandwidth consumption. The monitoring informs both enforcement decisions—blocking specific scrapers—and business decisions about licensing versus absorption.

Local news outlets publishing unique, irreplaceable content may command premium pricing in TollBit’s licensing marketplace. School closure information, municipal meeting coverage and community event details that appear nowhere else can’t be aggregated from multiple sources—giving publishers pricing power if AI companies want access.

Publishers can explore TollBit’s free monitoring at tollbit.com. Implementation takes under 30 minutes with no upfront costs or monthly fees. The platform delivers value through knowledge first, potential revenue later.

Frequently Asked Questions

The post TollBit tracks bot traffic Google Analytics misses appeared first on The Media Copilot.

Creative Commons has weighed in on one of the hottest debates in AI content licensing: Should websites charge AI companies to crawl their content?

The nonprofit published its position on pay-to-crawl systems last week. The verdict? Cautious support with major caveats.

Pay-to-crawl refers to technical systems that automate compensation when AI crawlers access web content. Think of it as a toll booth for bots.

Creative Commons sees potential benefits. These systems could help independent publishers generate revenue and manage server costs from heavy AI traffic. They might keep content publicly accessible that would otherwise vanish behind paywalls.

But the organization has serious concerns. Pay-to-crawl systems “could be cynically exploited by rights holders to generate excessive profits, at the expense of human access and without necessarily benefiting the original creators,” the group wrote.

The bigger worry: these systems could morph into something resembling digital rights management, “turning the web from a medium of sharing and remixing into a tightly monitored content delivery channel.”

Creative Commons proposed seven principles for responsible implementation. The highlights: Pay-to-crawl should not become a default setting imposed by web hosts. Systems should allow nuanced controls, not blanket rules. Researchers, nonprofits, and educators must retain access. And these systems should avoid surveillance architectures that track how content gets used downstream.

Why it matters for newsrooms: As publishers negotiate licensing deals with AI companies, pay-to-crawl could become another tool in the monetization toolkit. But poorly designed systems could block legitimate journalism uses like archiving, research, and fair use excerpting.

The organization is inviting feedback on its principles as the technology develops.

The post Creative Commons warns pay-to-crawl could become the new DRM appeared first on The Media Copilot.