Should your site serve Markdown to AI crawlers?
If you can, yes. A growing number of sites will hand an AI assistant a clean Markdown version of a page instead of the full HTML — less noise for the machine, and far less room for it to misread you.
What it actually is
When something requests a page, it can say what format it would prefer. A browser wants HTML. An AI assistant would much rather have the words. Markdown negotiation means the site notices that preference and answers with a stripped-back version: the headings, the paragraphs, the links, and none of the navigation, cookie banners, tracking scripts or layout scaffolding wrapped around them.
Why it matters
An assistant reading raw HTML has to work out which parts of the page are the content and which are furniture, and it does not always get that right. Serving Markdown removes the guesswork. In practice that means fewer summaries that quote your cookie notice, fewer that miss the point of the page, and a better chance the assistant repeats what you actually said rather than what it inferred from the wreckage.
What good looks like
A site that, when asked for a Markdown representation, returns one — with the real page content intact and the surrounding chrome gone. The Markdown should say the same thing the page says. A version that quietly drops half the article is worse than not offering one, because now the assistant is confidently working from a partial copy.
Being honest about it
This is an emerging capability rather than a settled standard, and most sites do not offer it. No agent requires it — one that asks for markdown and is handed HTML reads the HTML instead rather than giving up. On some platforms it is a genuine piece of infrastructure work rather than a setting. We flag it because the sites that do offer it get read more accurately, not because its absence is a problem you must solve this quarter.
Where most sites go wrong
The most common issue is not absence but a mismatch between what the platform advertises and what actually reaches a crawler. A layer in front of the site — a proxy, a CDN, a cache — can strip or ignore the preference, so the feature looks enabled from the inside and does nothing from the outside. The only way to know is to ask the way a crawler asks, from outside your own network, which is exactly what a scan does.
On the web
State of the Web · November 2026In November 2026, 4% of the sites we could read passed the “Clean content for agents (Markdown)” check.
Next.js sites pass it most often, at 13%. Shopify sites pass it least often, at 0%.
On the sites scanned in both October 2026 and November 2026, it went from 15% to 13%.
The month-by-month line counts the sites scanned in each calendar month, so it can differ from the figure above, which uses each site’s latest scan from the last three months.
Goes with
-
Sites that pass this are 12.5 times as likely to pass the “Content usage preferences declared” check too. 30% of the sites that pass the first check pass this one, against 2% of the sites that do not.
-
Sites that pass this are 6.5 times as likely to pass the “security.txt is published” check too. 23% of the sites that pass the first check pass this one, against 4% of the sites that do not.
-
Sites that pass this are 4 times as likely to pass the “Core security headers are set” check too. 26% of the sites that pass the first check pass this one, against 7% of the sites that do not.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call