Cloudflare today launched a feature that automatically converts HTML pages to markdown when AI agents request them, potentially changing how newsrooms publish content for AI-powered search and discovery.
What do 1,000 journalists and PR pros know about AI that you don't? They took AI Quick Start, a 1-hour live class from The Media Copilot. 94% satisfaction. Find out how to work smarter with AI in just 60 minutes. Get 20% off with the code AIPRO: https://mediacopilot.ai/
Key Takeaways
- Cloudflare’s Markdown for Agents converts HTML to markdown for AI agent requests.
- Conversion happens at the edge, no separate publisher pipeline needed.
- Could reshape AI-search publishing and centralize more decisions through Cloudflare.
The feature, called Markdown for Agents, uses content negotiation headers to detect when an AI system requests a page. When detected, Cloudflare’s network converts the HTML to markdown on the fly before serving it.
This matters because AI systems increasingly drive traffic to news sites, and those systems prefer markdown over HTML. The conversion happens automatically at the network edge, removing the need for publishers to maintain separate markdown versions of their content.
How it works
AI agents add an Accept: text/markdown header to their requests. Cloudflare detects this, fetches the original HTML from the origin server and converts it to markdown before returning it to the agent.
The company provides a simple curl example:
curl https://developers.cloudflare.com/page \
-H "Accept: text/markdown"Cloudflare includes an x-markdown-tokens header in responses that estimates token count, letting developers calculate context window sizes or plan chunking strategies.
Why markdown for AI
Markdown’s explicit structure makes it easier for AI systems to process. HTML pages have grown heavier over the years, making them harder to parse. AI agents typically filter out non-essential elements and scan for relevant content — a process markdown simplifies.
Most AI pipelines already convert HTML to markdown as a standard step, but this wastes computation, adds costs and processing complexity, and may not reflect how content creators intended their material to be used.
Cloudflare’s approach moves that conversion to the network edge, where it can happen efficiently at scale.
Content signals included
Converted responses include a Content-Signal header indicating how the content can be used:
Content-Signal: ai-train=yes, search=yes, ai-input=yesThis signals that content is available for AI training, search results and AI input (including agentic use). Cloudflare plans to add custom Content Signal policy options in the future.
The Content Signals framework launched during Cloudflare’s Birthday Week last year, letting publishers express preferences for how their content gets used after access.
Tracking usage
Cloudflare Radar now includes content type insights for AI bot and crawler traffic, showing the distribution of content types returned to AI agents grouped by MIME type category.
Publishers can filter requests for markdown by specific agents or crawlers. The data tracks how AI bots, crawlers and agents consume web content over time, accessible via public APIs and Cloudflare’s Data Explorer.
Availability
The feature is in beta at no cost for Pro, Business and Enterprise plans, plus SSL for SaaS customers. To enable it, log into the Cloudflare dashboard, select your account and zone, find Quick Actions and toggle Markdown for Agents.
Cloudflare already enabled the feature on its developer documentation and blog. Other conversion options include Workers AI’s AI.toMarkdown() method (supports multiple document types and summarization) and the Browser Rendering /markdown REST API (for dynamic pages requiring browser rendering before conversion).
Implementation considerations
For newsrooms using Cloudflare, this feature requires no code changes. The conversion happens automatically when AI agents request content with the appropriate headers.
Publishers not using Cloudflare can still convert documents using Cloudflare’s Workers AI or Browser Rendering APIs, though these require integration work.
The shift toward AI-driven discovery means newsrooms should consider how their content appears to AI systems, not just human readers. Markdown for Agents removes a technical barrier to that optimization.
