Bots read HTML just fine.

The pitch, and the problem

The idea sounds smart. AI bots have to strip menus, scripts and styling out of every page they read. So give them a clean markdown copy instead, and they'll read more of your site, more often.

Cloudflare, Vercel and a long list of plugins now make it easy. Some audit tools flag sites that skip it.

The real question is simpler: does it get you cited more? We pulled together the tests, the server logs and what the search engines have said. We also tried it on this site and broke something.

What the controlled test found

The best evidence so far comes from Profound, an AI visibility platform. In early 2026 it ran a randomized test across six websites and 381 pages for three weeks.

Half the pages served bots a clean markdown copy. The other half served normal HTML. People always saw HTML.

Profound's own read: no clear evidence that markdown increases bot traffic, and any effect is likely small. AI models already parse HTML well.

Two caveats matter. The test could only detect effects bigger than about 40 percent. And bot traffic swung so much that it rose 12 percent before any markdown was served.

So a small lift is possible. A big one would have shown up.

The token math is real, for the reader

Markdown is much smaller than HTML. That part is not in dispute.

But look at who saves. Fewer tokens means a cheaper, faster read for the agent. Your hosting bill barely moves, and nothing so far shows that a lighter page gets picked as a source more often.

Where it does help is long technical pages read by coding agents, which work inside tight context limits. Docs, API references and pricing tables are good candidates.

Who actually asks for markdown

An agent that wants markdown can say so in its request, with an Accept: text/markdown header. In February 2026, Checkly logged what popular agents actually send.

The pattern is clear. Coding agents want markdown and ask for it. General assistants mostly prefer HTML or take whatever comes back.

That means you don't need to guess who should get markdown. The agents that want it tell you.

llms.txt: published everywhere, read almost nowhere

The llms.txt file is a markdown map of your site, placed at the root for AI tools. It spread fast. Server logs show very few bots read it.

• Ahrefs, 137,210 sites, May 2026. 28 percent publish an llms.txt. 97 percent of those files got zero requests that month. Named AI bots made under a fifth of the requests that did happen, led by GPTBot and Claude Code.
• Otterly, 90-day log test. 84 of more than 62,100 AI bot visits touched llms.txt, about 0.1 percent.
• SE Ranking, about 300,000 sites. No clear link between having the file and being cited. Only 1 of the 50 most-cited domains had one.

It costs almost nothing to publish, and our llms.txt generator builds one in a minute. Just don't expect it to change your AI visibility on its own.

What Google and Bing say

In February 2026, both search engines answered a question about separate markdown pages for AI.

Google's John Mueller asked, "Why would they want to see a page that no user sees?" He added that LLMs already read normal HTML well.

Bing's Fabrice Canel said they would crawl the regular page anyway to check that it matches. He noted that pages only bots see are often neglected and broken.

In June 2026, Google added that markdown and machine-readable AI files aren't needed to appear in Search, and that having one neither helps nor hurts.

None of this bans markdown. It does mean a bot-only version of your site carries risk and no proven upside in search.

Three ways to serve markdown, ranked

There are three common ways to hand bots markdown. They are not equally safe.

• 01Accept negotiationRecommended. Markdown goes only to requests that ask for it firstReachesClaude Code, Cursor, OpenCode and future agents that askRiskLow. Send Vary: Accept so caches keep versions apart
• 02Declared .md copySafe extra. A second URL linked from the page and llms.txtReachesAgents that follow link tags or llms.txtRiskLow, with noindex and a canonical back to the page
• 03User-agent swapAvoid. Markdown goes to a list of bot namesReachesOnly the bots on your listRiskHigh. Fetchers can reject it, and CDNs can mix up versions

The user-agent swap is the one most plugins and guides recommend. It is also the one that breaks without telling you.

What happened when we tried it

This site used to swap in markdown for a list of AI bot names, including ChatGPT-User and PerplexityBot. In October 2026 we asked ChatGPT to summarize one of our guides.

It couldn't. It reported a markdown content-type error and asked us to paste the article instead. Perplexity also failed to read the page. Our own readiness checker had shown the warning signs: some checks came back refused for GPTBot even though a normal browser loaded the page.

We switched every bot to the same HTML people see. Markdown is now served only to agents that ask for it first, plus the declared index.md copy.

Other sites have served ChatGPT-User markdown without trouble, including in Profound's test. That's the point. Each fetcher handles content types its own way and can change without notice. A swap that works today can fail quietly tomorrow, and you only find out when someone tests it.

The setup we recommend

Give every bot the same HTML. Offer markdown to agents that ask for it, and make the copy easy to find.

1. Keep your HTML clean. Main content in the page, not loaded by scripts. This is what every bot reads.
2. Answer Accept: text/markdown. Serve markdown only when the request lists it first, and send Vary: Accept so caches keep the versions apart.
3. Publish a static .md copy. Link it with <link rel="alternate" type="text/markdown"> and list it in llms.txt.
4. Keep the copy out of search. Send X-Robots-Tag: noindex and a canonical header that points back to the HTML page.
5. Don't switch by bot name. Drop any rule that serves a different version based on the user agent.

On Apache, the negotiation rule fits in a few lines of .htaccess:

RewriteCond %{HTTP_ACCEPT} ^\s*text/(x-)?markdown [NC]
RewriteCond %{DOCUMENT_ROOT}/$1index.md -f
RewriteRule ^(.*/)?$ /$1index.md [L]

On Cloudflare, the Markdown for Agents setting does the conversion for you when an agent asks.

What actually moves AI visibility

Markdown is a convenience for agents. These are the things that decide whether AI tools find and cite you.

• Access. AI bots allowed in robots.txt and not blocked by your firewall or CDN.
• Search indexing. ChatGPT and Perplexity lean on search results, so pages need to be indexed in Google and Bing.
• Quotable content. Clear answers, real numbers and short sections an AI can lift into a reply.
• Mentions elsewhere. Reviews, directories and articles that name your brand.

Our GEO guide covers each of these in depth.

A markdown checklist

• Every bot, including ChatGPT-User and PerplexityBot, gets the same HTML as a browser.
• Requests that list text/markdown first get markdown, with a Vary: Accept header.
• Each key page has a .md copy declared with a link tag and listed in llms.txt.
• The .md copies send noindex and a canonical header back to the HTML.
• The markdown matches the HTML, and gets updated when the page does.
• You test a real AI tool on a real page after every change.

Our free AI agent readiness checker tests all three markdown methods on any page and flags user-agent swaps.

Sources

• Profound: Does markdown increase AI bot traffic?
• Checkly: The current state of content negotiation for AI agents
• PPC Land: Cloudflare cuts AI token costs by 80% with markdown conversion
• Ahrefs: 97% of llms.txt files never get read
• Optimizely: Does llms.txt work?
• PPC Land: Google and Bing on separate markdown pages for AI
• Vercel: Make your documentation readable by AI agents

Frequently asked questions