What is llms.txt, and does your site need one?
A plain-text file at the root of your site that gives machines a curated map of your most important pages. It is a community proposal (llmstxt.org), not a ratified standard. Anything that reads text can read it and publishing one costs you nothing, but we cannot tell you which assistants read it.
What it actually is
Think of it as a menu written for machines. Your sitemap lists everything you have ever published; llms.txt says "here are the pages that actually matter, and here is what each one is for." When an AI assistant tries to understand your business in a few seconds, it is the difference between handing someone a filing cabinet and handing them a one-page brief.
Why it matters
AI answer engines do not crawl the way Google does — they read a little and infer a lot. If the first things they find are your cookie policy and a paginated blog archive, that is the impression they form of your business. A good llms.txt puts your services, your pricing and your best explainers at the front of the queue, so the summary an assistant gives about you is the one you would have written yourself.
What good looks like
A file at yoursite.com/llms.txt that opens with your site name, gives a short plain-English summary of what you do, then groups your key links under simple headings — services, docs, about — with a line of description beside each. A file with the structure the proposal describes, an H1 naming the site then sections of links, is one a machine can parse.
Being honest about it
It is a convention rather than a law, and nobody should promise you that a text file alone will get you cited. The honest position is that adoption is running ahead of any evidence of consumption: many sites publish one, and the proposal itself says only that agents can use it on demand. We flag its absence anyway, because we score whether a machine can find a curated map of your content, and this is the one file whose only job is to be that map. It is free to publish, and it is the one place you get to say, in your own words, which of your pages matter.
Where most sites go wrong
Two failure modes. Either there is nothing there at all, or — more commonly, and worse — there is a file that exists but is thin, stale, or points at pages that have since moved. A stub with a heading and no links helps nobody. A dump of every URL you own defeats the entire point, because curation was the value. And an llms.txt written a year ago quietly tells an AI the wrong story about what your business does today.
On the web
State of the Web · November 2026In November 2026, 28% of the sites we could read passed the “llms.txt is present and valid” check.
Shopify sites pass it most often, at 92%. Webflow sites pass it least often, at 26%.
On the sites scanned in both August 2026 and November 2026, it went from 32% to 35%.
The month-by-month line counts the sites scanned in each calendar month, so it can differ from the figure above, which uses each site’s latest scan from the last three months.
Goes with
-
Sites that pass this are 7.5 times as likely to pass the “Clean content for agents (Markdown)” check too. 11% of the sites that pass the first check pass this one, against 1% of the sites that do not.
-
Sites that pass this are 7 times as likely to pass the “Content usage preferences declared” check too. 10% of the sites that pass the first check pass this one, against 1% of the sites that do not.
-
Sites that pass this are 4 times as likely to pass the “security.txt is published” check too. 9% of the sites that pass the first check pass this one, against 2% of the sites that do not.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call