Meta’s crawlers hit websites 9.1 billion times in the second quarter of 2026 and sent almost nobody back in return. That single figure, pulled from DataDome’s Q2 2026 AI Traffic Report, captures the widening split between the agents that consume publisher infrastructure and the ones that actually deliver readers.
DataDome’s network processed 17.7 billion AI agent requests between April and June, a 45% jump from Q1’s 12.2 billion. The report draws on 5 trillion signals analyzed daily across more than 400 enterprises. Since January, the network has logged over 30 billion AI agent requests in total, and the monthly curve kept climbing: 4.77 billion in April, 6.29 billion in May, 6.60 billion in June.
Meta drove most of that growth. Its two crawlers do different jobs. Meta-ExternalAgent reads websites to train AI models without sending traffic or compensation back to publishers. Meta-WebIndexer works more like Google’s crawler, indexing pages so Meta AI can answer real-time queries. In Q2, ExternalAgent grew 74% to 5.3 billion requests and WebIndexer grew 163% to 3.75 billion. In June, WebIndexer passed ExternalAgent in monthly volume for the first time, a sign Meta is investing in the answering side of AI as much as the training side.
Crawl volume and referral value are pulling apart. ChatGPT-User, the top agent in Q1, fetched pages 6% less often in Q2. Yet OpenAI’s chatbot still commands 80% to 88% of all AI-driven referral traffic and grew referrals 17% quarter over quarter. Among the rest, Claude referrals more than doubled to 876,000, Perplexity grew 37%, and Grok collapsed 74% to just 24,000 visits.
The distinction matters because publishers have spent the past two years arguing that AI companies use their content without returning referral traffic or other value in exchange. That tension runs through the AI scraping economy, and DataDome’s numbers put figures on it.
Sponsored. AI Bot Access Check by oasy.ai: check any site's AI bot policy live: which AI crawlers and agents (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) its robots.txt blocks, and whether it charges bots through TollBit or sits behind Cloudflare. Free, no key: https://oasy.ai/api/aibots/4j52vn2h?domain=nytimes.com
The report also flags a new signal worth watching: Model Context Protocol traffic. MCP, the connective layer between AI agents and external tools that Anthropic introduced in late 2024, went from negligible volume to peaks near 500,000 requests a day. Most requests come from AI agents taking inventory through calls such as initialize, tools/list and prompts/list. Rather than reading content, those requests reveal what an agent intends to do before it takes action.
For newsrooms and publishers, the practical message is that bot-or-not detection no longer cuts it. Meta-ExternalAgent, Meta-WebIndexer, ChatGPT-User and a chat session all demand different responses. DataDome recommends agent-level classification, MCP monitoring, and identity validation through standards like Web Bot Auth rather than trusting user-agent strings, which are easily spoofed. Any allowlist granting automatic access based on a trusted agent name is exposed.
Already, 54% of DataDome customers have adopted agent trust policies. The firm frames that as a leading indicator. As Q3 data arrives, the open question is whether Meta’s tilt toward real-time indexing holds, and whether publishers can tell the difference between an agent burning their bandwidth and one bringing them an audience.
