--- title: "Markdown for AI Bots: What the Data Says | Agentic Organic" description: "Should you serve AI bots markdown? What tests, server logs and Google say about markdown, llms.txt and content negotiation, plus the setup that works." url: https://agenticorganic.com/latent/agentic-search/markdown-for-ai-bots/ last_updated: 2026-10-18 --- [Home](https://agenticorganic.com/) / [Latent](https://agenticorganic.com/latent/) / [Agentic search](https://agenticorganic.com/latent/agentic-search/) # Bots read HTML just fine. Serving AI bots a markdown copy of every page is the GEO tactic of 2026. The tests, the logs and the search engines all point the same way: offer markdown, but don't force it. Updated Oct 18, 2026 · 7 min read · By Agentic Organic ## Key takeaways - In the largest controlled test so far, serving bots markdown made no clear difference to bot traffic. - Markdown cuts tokens by 80 to 99 percent. That saves the agent money, not you. - Coding agents like Claude Code and Cursor ask for markdown with an Accept header. Most other fetchers prefer HTML. - 97 percent of llms.txt files got zero requests in a month. Publish one, but don't expect it to move citations. - Serve every bot the same HTML, and give markdown to agents that ask for it first. ## The pitch, and the problem The idea sounds smart. AI bots have to strip menus, scripts and styling out of every page they read. So give them a clean markdown copy instead, and they'll read more of your site, more often. Cloudflare, Vercel and a long list of plugins now make it easy. Some audit tools flag sites that skip it. The real question is simpler: does it get you cited more? We pulled together the tests, the server logs and what the search engines have said. We also tried it on this site and broke something. ## What the controlled test found The best evidence so far comes from Profound, an AI visibility platform. In early 2026 it ran a randomized test across six websites and 381 pages for three weeks. Half the pages served bots a clean markdown copy. The other half served normal HTML. People always saw HTML. | Measure | Result | |---|---| | Median change per page | About one extra bot visit | | Average change | About 16 percent higher, within random noise | | ChatGPT-User, 73% of bot traffic | About 20 percent higher, not statistically significant | | Citations | Not measured | Profound's own read: no clear evidence that markdown increases bot traffic, and any effect is likely small. AI models already parse HTML well. Two caveats matter. The test could only detect effects bigger than about 40 percent. And bot traffic swung so much that it rose 12 percent before any markdown was served. So a small lift is possible. A big one would have shown up. ## The token math is real, for the reader Markdown is much smaller than HTML. That part is not in dispute. | Source | HTML | Markdown | Cut | |---|---|---|---| | Cloudflare, sample blog post | 16,180 tokens | 3,150 tokens | About 80% | | Checkly, docs page | 180,573 tokens | 478 tokens | 99.7% | | Vercel, docs page | 500 KB | 2 KB | 99.6% | But look at who saves. Fewer tokens means a cheaper, faster read for the agent. Your hosting bill barely moves, and nothing so far shows that a lighter page gets picked as a source more often. Where it does help is long technical pages read by coding agents, which work inside tight context limits. Docs, API references and pricing tables are good candidates. ## Who actually asks for markdown An agent that wants markdown can say so in its request, with an `Accept: text/markdown` header. In February 2026, Checkly logged what popular agents actually send. | Agent | What it asks for first | |---|---| | Claude Code | Markdown | | Cursor | Markdown | | OpenCode | Markdown | | OpenAI Codex | HTML | | GitHub Copilot | HTML | | Gemini CLI | Anything | | Windsurf | Anything | The pattern is clear. Coding agents want markdown and ask for it. General assistants mostly prefer HTML or take whatever comes back. That means you don't need to guess who should get markdown. The agents that want it tell you. ## llms.txt: published everywhere, read almost nowhere The llms.txt file is a markdown map of your site, placed at the root for AI tools. It spread fast. Server logs show very few bots read it. - **Ahrefs, 137,210 sites, May 2026.** 28 percent publish an llms.txt. 97 percent of those files got zero requests that month. Named AI bots made under a fifth of the requests that did happen, led by GPTBot and Claude Code. - **Otterly, 90-day log test.** 84 of more than 62,100 AI bot visits touched llms.txt, about 0.1 percent. - **SE Ranking, about 300,000 sites.** No clear link between having the file and being cited. Only 1 of the 50 most-cited domains had one. It costs almost nothing to publish, and our [llms.txt generator](https://agenticorganic.com/tools/llms-txt-generator/) builds one in a minute. Just don't expect it to change your AI visibility on its own. ## What Google and Bing say In February 2026, both search engines answered a question about separate markdown pages for AI. Google's John Mueller asked, "Why would they want to see a page that no user sees?" He added that LLMs already read normal HTML well. Bing's Fabrice Canel said they would crawl the regular page anyway to check that it matches. He noted that pages only bots see are often neglected and broken. In June 2026, Google added that markdown and machine-readable AI files aren't needed to appear in Search, and that having one neither helps nor hurts. None of this bans markdown. It does mean a bot-only version of your site carries risk and no proven upside in search. ## Three ways to serve markdown, ranked There are three common ways to hand bots markdown. They are not equally safe. - **Accept negotiation (recommended)**: Markdown goes only to requests that ask for it first · Reaches: Claude Code, Cursor, OpenCode and future agents that ask · Risk: Low. Send Vary: Accept so caches keep versions apart - **Declared .md copy (safe extra)**: A second URL linked from the page and llms.txt · Reaches: Agents that follow link tags or llms.txt · Risk: Low, with noindex and a canonical back to the page - **User-agent swap (avoid)**: Markdown goes to a list of bot names · Reaches: Only the bots on your list · Risk: High. Fetchers can reject it, and CDNs can mix up versions The user-agent swap is the one most plugins and guides recommend. It is also the one that breaks without telling you. ## What happened when we tried it This site used to swap in markdown for a list of AI bot names, including ChatGPT-User and PerplexityBot. In October 2026 we asked ChatGPT to summarize one of our guides. It couldn't. It reported a markdown content-type error and asked us to paste the article instead. Perplexity also failed to read the page. Our own readiness checker had shown the warning signs: some checks came back refused for GPTBot even though a normal browser loaded the page. We switched every bot to the same HTML people see. Markdown is now served only to agents that ask for it first, plus the declared index.md copy. Other sites have served ChatGPT-User markdown without trouble, including in Profound's test. That's the point. Each fetcher handles content types its own way and can change without notice. A swap that works today can fail quietly tomorrow, and you only find out when someone tests it. What do AI bots get from your site? The free checker requests your page as nine AI bots and tests all three markdown methods. [Run the free checker](https://agenticorganic.com/tools/agent-readiness/) ## The setup we recommend Give every bot the same HTML. Offer markdown to agents that ask for it, and make the copy easy to find. 1. **Keep your HTML clean.** Main content in the page, not loaded by scripts. This is what every bot reads. 2. **Answer Accept: text/markdown.** Serve markdown only when the request lists it first, and send `Vary: Accept` so caches keep the versions apart. 3. **Publish a static .md copy.** Link it with `` and list it in llms.txt. 4. **Keep the copy out of search.** Send `X-Robots-Tag: noindex` and a canonical header that points back to the HTML page. 5. **Don't switch by bot name.** Drop any rule that serves a different version based on the user agent. On Apache, the negotiation rule fits in a few lines of `.htaccess`: ``` RewriteCond %{HTTP_ACCEPT} ^\s*text/(x-)?markdown [NC] RewriteCond %{DOCUMENT_ROOT}/$1index.md -f RewriteRule ^(.*/)?$ /$1index.md [L] ``` On Cloudflare, the Markdown for Agents setting does the conversion for you when an agent asks. ## What actually moves AI visibility Markdown is a convenience for agents. These are the things that decide whether AI tools find and cite you. - **Access.** AI bots allowed in robots.txt and not blocked by your firewall or CDN. - **Search indexing.** ChatGPT and Perplexity lean on search results, so pages need to be indexed in Google and Bing. - **Quotable content.** Clear answers, real numbers and short sections an AI can lift into a reply. - **Mentions elsewhere.** Reviews, directories and articles that name your brand. Our [GEO guide](https://agenticorganic.com/latent/geo/generative-engine-optimization-guide/) covers each of these in depth. ## A markdown checklist - Every bot, including ChatGPT-User and PerplexityBot, gets the same HTML as a browser. - Requests that list text/markdown first get markdown, with a `Vary: Accept` header. - Each key page has a .md copy declared with a link tag and listed in llms.txt. - The .md copies send noindex and a canonical header back to the HTML. - The markdown matches the HTML, and gets updated when the page does. - You test a real AI tool on a real page after every change. Our free [AI agent readiness checker](https://agenticorganic.com/tools/agent-readiness/) tests all three markdown methods on any page and flags user-agent swaps. ## Sources - [Profound: Does markdown increase AI bot traffic?](https://www.tryprofound.com/blog/does-markdown-increase-ai-bot-traffic) - [Checkly: The current state of content negotiation for AI agents](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/) - [PPC Land: Cloudflare cuts AI token costs by 80% with markdown conversion](https://ppc.land/cloudflare-cuts-ai-token-costs-by-80-with-markdown-conversion/) - [Ahrefs: 97% of llms.txt files never get read](https://ahrefs.com/blog/llmstxt-study/) - [Optimizely: Does llms.txt work?](https://www.optimizely.com/de/field-notes/articles/does-llms-txt-work) - [PPC Land: Google and Bing on separate markdown pages for AI](https://ppc.land/google-and-bing-say-no-separate-markdown-pages-for-ai-violate-search-policies/) - [Vercel: Make your documentation readable by AI agents](https://vercel.com/kb/guide/make-your-documentation-readable-by-ai-agents) ## Frequently asked questions ### Will a markdown version of my site get me cited by ChatGPT? There is no evidence it will. The largest controlled test found no clear lift in bot traffic, and no study so far has tied markdown to more citations. Access, search indexing and content worth quoting matter more. ### Is serving markdown to bots cloaking? It can look like it. Google and Bing both questioned pages that only bots see. Serving markdown when an agent asks for it with an Accept header, and keeping it identical to the HTML, keeps you on the safe side. ### Do I need an llms.txt file? It is cheap to publish and does no harm, but most files are never requested. Treat it as a map for coding agents and tools, not a ranking factor. ### What is the easiest way to offer markdown? If you use Cloudflare, its Markdown for Agents setting converts pages when an agent asks for markdown. On other hosts, publish an index.md next to each page and serve it to requests that list text/markdown first. ## More from Latent [All categories](https://agenticorganic.com/latent/) ### [Can a bot buy from you?](https://agenticorganic.com/latent/agentic-search/agentic-search-guide/) AI agents don't just answer questions anymore. 8 min read ### [Google reads you first.](https://agenticorganic.com/latent/seo/seo-foundations-guide/) Before a customer sees your site, Google has to find it, read it and decide it deserves a spot. 9 min read ### [Be the answer AI gives.](https://agenticorganic.com/latent/geo/generative-engine-optimization-guide/) When a buyer asks ChatGPT for a recommendation, the answer names three or four businesses. 8 min read ### [Your first agent should be boring.](https://agenticorganic.com/latent/ai-agents/ai-agents-business-guide/) The AI agents that pay for themselves rarely make headlines. 8 min read