• Skip to main content
  • Skip to header right navigation
  • Skip to site footer
The Media Copilot

The Media Copilot

How AI is changing Media, journalism and content creation

  • News
  • Reviews
  • Guides
  • AI Courses
    • AI Quick Start
    • NEW—AI for Media
    • Custom AI Training for Teams
  • Newsletter
  • Podcast
  • Events
    • GEO Dinner Series
    • Webinars
  • About
    • Careers

AI Web Scraping: The Invisible Threat to Websites

Digital Trends used TollBit to track AI bot scraping, revealing how AI overviews erode search traffic.

Person's finger touching a glowing red digital "Stop" sign hologram
AI web scrapers are gutting sites' web traffic. Tollbit helps quantify what kind of data is most likely to be extracted. (Credit: LORD)
Mar 3, 2026

By The Copilot , generated from Inside Digital Trends’ selection of Tollbit to help monitor AI traffic to its website by Z. Waite  on January 5, 2026

Dan Gaul, the co-founder and chief technical officer of Digital Trends, suspected AI was gutting his site’s search traffic. Traditional analytics showed referrals dropping as AI overviews replaced direct links. But Google Analytics tracks human visitors, not the bots harvesting content at scale. Gaul couldn’t quantify the extraction.

Key Takeaways

  • Digital Trends used TollBit to expose 4.1M bot scrapes against 4,200 chatbot referrals.
  • ChatGPT alone accounted for 87.8% of automated traffic.
  • A 30-minute setup gave Digital Trends data to renegotiate content economics.

TollBit’s free monitoring revealed the brutal math: 4.1 million bot scrapes in one week delivered only 4,200 human referrals. ChatGPT alone accounted for 87.8 percent of automated traffic. The data doesn’t generate revenue yet, but it quantifies what was invisible.

“The value of TollBit is the knowledge,” Gaul says.

The gist

Digital Trends’ lightweight implementation exposed the AI extraction economy:

  • Implementation required under 30 minutes with no upfront costs
  • Analytics revealed 966-to-1 scrape-to-referral ratio documenting traffic asymmetry
  • Monitoring identified content types facing maximum AI substitution

How they did it

Digital Trends’ implementation prioritized understanding bot patterns before enforcement:

  • Installed JavaScript tracking tag: Added lightweight monitoring code similar to Google Analytics, requiring no backend access or complex technical integration.
  • Configured DNS settings: Completed minimal DNS configuration enabling TollBit to route detected bots appropriately—implementation handled by any IT team in under 30 minutes.
  • Established baseline metrics: Monitored which AI services accessed content, scraping frequency, specific pages targeted and human referral ratios revealing value exchange.
  • Analyzed content patterns: Used data to identify which article types faced maximum extraction—evergreen how-tos and spec-heavy reviews versus original analysis and narrative features.
  • Informed editorial strategy: Applied monitoring insights to content planning, recognizing spec-centric information faces AI substitution while creative writing retains unique value.
  • Subscribe to our newsletter

    Sponsored. Journalists, PR pros and communicators: the fall cohort of AI for Media starts October 13, six live Tuesday sessions with Pete Pachal plus two 1:1 coaching calls. Code AISEARCH500 takes $500 off the $1,500 price for anyone who found the course through AI search, a bigger discount than is offered anywhere else.

    How AI is changing media, journalism, and content creation.

    Learn More

Key numbers

  • Implementation time: Under 30 minutes for monitoring setup
  • Cost: Free for publishers (no upfront fees or monthly subscription)
  • Bot scrapes tracked: 4.1 million in one week
  • Human referrals received: 4,200 in same period
  • Scrape-to-referral ratio: 976 to 1
  • ChatGPT dominance: 87.8 percent of all AI bot traffic (3.6 million monthly scrapes)
  • Revenue generated: None—monitoring only, monetization not activated

What to watch for

Implementation considerations and realistic expectations:

  • No guaranteed revenue: Licensing marketplace remains nascent—monetization is speculative positioning for future opportunities, not immediate income stream.
  • Requires ongoing monitoring: Dashboard review and policy development demand attention—free implementation doesn’t mean zero time investment.
  • Misconfiguration risks: Accidentally blocking legitimate search engines or accessibility tools could harm SEO—allowlist maintenance requires vigilance.
  • Bot spoofing continues: Unauthorized scrapers masquerade as browsers—detection requires ongoing adaptation as evasion tactics evolve.

Publishers can explore TollBit’s free monitoring at tollbit.com. The platform delivers value through visibility into bot behavior, with potential revenue dependent on AI licensing marketplace maturation.

Posts co-authored by The Copilot are drafted with AI and then carefully edited by Media Copilot editors. Our AI-assisted process allows us to bring more valuable content to our readers while preserving accuracy and quality.

Contributors

  • Z. Waite: Author

    Z. Waite is a journalist, researcher, and current graduate student at the UC Berkeley School of Journalism, where they report on artificial intelligence and study the impact of new technologies on the news industry.

  • The Copilot: Coauthor

    I'm a generative AI writer for The Media Copilot. I help author posts, and with the help of human editors, play a growing role in the site's content strategy.

  • Christopher Allbritton: Editor

    Christopher Allbritton covers AI adoption in journalism and newsroom transformation. He brings 20+ years of journalism experience, including roles as Reuters' Pakistan Bureau Chief and TIME's Middle East Correspondent.

Category: Guides, How-toTags:quick reference| monetization| tollbit| traffic analysis| bot blocking
Share this post:
FacebookTweetLinkedInEmail
  • Related articles

Footballco credits AI assistant Goal-e for 1bn World Cup page views

Read moreFootballco credits AI assistant Goal-e for 1bn World Cup page views

Publishers are fighting the wrong battle against AI

Read morePublishers are fighting the wrong battle against AI

Axios publisher says SEO is ‘not the answer anymore’ as AI search grows

Read moreAxios publisher says SEO is ‘not the answer anymore’ as AI search grows

IAB drafts AI advertising measurement framework as agents replace human clicks

Read moreIAB drafts AI advertising measurement framework as agents replace human clicks

Thomson Reuters says it spent $40 million to build its own frontier AI model

Read moreThomson Reuters says it spent $40 million to build its own frontier AI model

Current top 10 US news sites lost 32% of traffic from 2024 peak

Read moreCurrent top 10 US news sites lost 32% of traffic from 2024 peak

The Media Copilot

The Media Copilot is an independent media organization covering the intersection of AI and media. Founded by journalist Pete Pachal, we produce journalism, analysis, and courses meant to help newsrooms and PR professionals navigate the growing presence of AI in our media ecosystem.

  • LinkedIn
  • X
  • YouTube
  • Instagram
  • TikTok
  • Bluesky
  • About The Media Copilot
  • Careers
  • Advertising & Sponsorships
  • Our Methodology
  • Privacy Policy
  • Membership
  • Newsletter
  • Podcast
  • Contact

© 2026 · All Rights Reserved · Powered by Springwire.ai · RSS