The Internet is undergoing a fundamental shift — and two of its largest infrastructure companies want to be the ones who set the rules for it.
Cloudflare and GoDaddy announced a strategic partnership Tuesday designed to give creators and small businesses more visibility and control over how AI agents access their content. The centerpiece: GoDaddy is integrating Cloudflare’s AI Crawl Control directly into its website hosting platform, letting site owners allow, block, or signal that payment is required before any AI crawler touches their content. The companies are also supporting open standards — specifically GoDaddy’s own Agent Name Service (ANS) and Cloudflare’s Web Bot Auth — that would give AI agents a cryptographically verified identity rather than operating anonymously across the web.
The problem they’re solving is real and growing. AI agents are increasingly browsing, summarizing, and retrieving content on behalf of users — but most operate without identifying themselves, leaving site owners with no record of who scraped what and no mechanism to negotiate terms. “There needs to be a way to ensure that businesses and creators have the tools to easily identify, manage, and trust AI traffic,” the companies said in a joint statement.
ANS, which GoDaddy introduced as a global open standard, uses DNS and public key infrastructure to assign verifiable identities to AI agents — a layer of trust that lets site owners distinguish legitimate agents from impersonators. Cloudflare, which introduced Web Bot Auth in 2025 along with a Signature Agent Card developers can use to share their agent’s stated purpose, is supporting the standard alongside its own tools. The broader goal is a permission-based model for the agentic web — and a technical foundation for a fair value exchange in what both companies call the Answer Engine era. It’s a dynamic that mirrors broader tensions: as AI companies negotiate directly with publishers over content access, the underlying question of who controls and monetizes AI’s use of the web is far from settled.
Sponsored. Journalists, PR pros and communicators: the fall cohort of AI for Media starts October 13, six live Tuesday sessions with Pete Pachal plus two 1:1 coaching calls. Code AISEARCH500 takes $500 off the $1,500 price for anyone who found the course through AI search, a bigger discount than is offered anywhere else.
“The Internet is evolving into a high-velocity, AI-driven ecosystem, and that requires a new kind of transparent infrastructure,” said Stephanie Cohen, Chief Strategy Officer at Cloudflare. “By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model.”
For publishers and creators watching their search traffic erode as AI answers replace clicks, the partnership is a concrete — if early — signal that the infrastructure layer is beginning to take shape. Whether it scales to the full open web remains an open question. Cloudflare has already been converting web pages to markdown for AI agents, and the battle over how AI crawlers pay for content is accelerating.
“We move at the speed of the Internet,” said GoDaddy Chief Strategy Officer Jared Sine, “and we’re working with the broader industry to ensure the agentic open web does too.”
