Your website can serve clean Markdown to AI bots instead of making them dig through your HTML, and this site has been doing it for a while.
Try it yourself right now:
curl https://www.kevinleary.net/blog/
-H "Accept: text/markdown"
You’ll get this page back as structured Markdown with frontmatter, not HTML. Here’s what that approach is, why it’s worth doing, and how Cloudflare makes it a one-toggle feature for an entire site.
The Approach: Content Negotiation for Machines
Markdown has quickly become the lingua franca of AI systems. Every major assistant thinks in it, agent frameworks pass it around internally, and when an LLM ingests a web page, the first thing most pipelines do is strip the HTML down to something Markdown-shaped anyway.
The idea is to skip that step and serve the clean version directly, using a mechanism HTTP has had since the 1990s: content negotiation. A client says what format it prefers in the Accept header, and the server responds accordingly. Browsers ask for text/html and get the full page. An AI crawler or agent that asks for text/markdown gets the same content as Markdown: headings, lists, code blocks and links intact, with none of the nav, scripts, tracking or layout markup.
Same URL, same content, two representations. Humans see the designed page, machines get the structured text.
Why It’s Worth Doing
Tokens are the currency of AI, and HTML wastes them. A typical rendered page is 80-90% markup, scripts and boilerplate by weight. When an assistant retrieves your page to answer a question, everything it ingests competes for a limited context window. Serve Markdown and your actual content occupies the space your div soup was wasting, which means more of your page survives into the model’s working context.
Structure survives extraction. When an AI pipeline strips HTML itself, things get lost or mangled: table structure, code block boundaries, list nesting, which heading owns which paragraph. Markdown hands the model your content with the hierarchy explicit. Cleaner input produces more accurate answers about your content, with your headings intact as retrievable anchors.
You control the canonical machine version. Rather than hoping every crawler’s HTML-cleaning heuristics treat you well, you decide what the machine-readable representation of your site is. That’s the same philosophy behind the structured /llm-api/ endpoint I run here and recommend in place of llms.txt: give machines typed, deliberate data instead of prose to interpret.
The audience is already here. This isn’t preparation for a future that might arrive. Cloudflare’s AI Crawl Control on this site logged about 16,000 requests from AI crawlers in the last seven days alone: ByteDance, Microsoft, OpenAI, Google, Apple, Anthropic, Meta, Perplexity and Amazon, all actively pulling content. AI search referrals are the fastest growing acquisition channel I track, and I’ve written about how that traffic gets earned. Serving those crawlers a better representation of your content is optimizing for a channel that’s already producing.
The Cloudflare Feature: Markdown for Agents
You could build this yourself with a converter and some origin logic, but if your site is behind Cloudflare there’s no reason to: a feature called Markdown for Agents does it at the edge for your entire site, with a single toggle.
That’s the whole setup, and it’s exactly what it looks like on this site’s dashboard: enabled, with 100% of markdown requests fulfilled. You’ll find it in the Cloudflare dashboard under AI Crawl Control on your domain’s overview. Flip it on and Cloudflare’s network converts any HTML page to Markdown on the fly whenever a request carries the Accept: text/markdown header.
What makes this genuinely seamless:
- Site-wide, instantly. Every page, post and template is covered the moment you enable it. No plugins, no origin changes, no per-page work, no separate URLs to maintain.
- Conversion happens at the edge. Your origin serves HTML exactly as it always has; Cloudflare transforms the response in transit. Your WordPress install doesn’t know the feature exists.
- Nothing changes for humans. Browsers never send the header, so regular visitors get the normal site. There’s no cloaking concern either, since it’s the same content in a different representation, delivered through a standard HTTP mechanism.
- It’s free on Pro plans and up. No add-on pricing, no usage billing. The only real limitation is that origin responses over 2 MB aren’t converted, which no well-built page should be hitting anyway.
Cloudflare has enabled the same feature on its own developer docs and blog, which tells you how they expect the ecosystem to move.
An Honest Caveat on Formatting
One finding from the 2026 research deserves a mention here, because it tempers the case. In a controlled experiment of 252,000 citation trials, structured versus dense formatting and organized versus scattered information made no reliable difference to which source a model cited. When two pages carry the same information, the better-formatted one does not win.
That does not undo the argument for serving Markdown, but it does narrow it. The gains here are about token efficiency, extraction fidelity and speed, all of which affect how much of your page gets consumed and how reliably. They are not a substitute for the things that decide citations, which are price, specifications, evidence, recency and retrieval position. Serve Markdown because it makes your content cheaper and cleaner to ingest, not because prettier structure wins the citation.
Where This Fits in an AEO Strategy
Markdown for Agents is a supporting player, not the whole game. The content still has to be worth citing, the fundamentals still decide the outcome, and structured data in your templates still hands machines facts that prose can’t. But it’s the rare optimization with this profile: one toggle, zero maintenance, zero risk to human visitors, and a measurably better representation of your site handed to every AI system polite enough to ask for it.
Most sites behind Cloudflare could turn this on today and simply haven’t heard of it. Now you have. It’s part of the standard setup I deploy for AEO and SEO consulting clients on Cloudflare, alongside the crawler analytics that show you exactly which AI systems are reading you, and how often they send humans back.
Supporting research: Vishwakarma, Kumar and Jamidar (2026), What Gets Cited: Competitive GEO in AI Answer Engines, SIGIR ’26, and LLM Ranking Factors 2026 from The Digital Bloom. I break both studies down in AI Ranking Factors.