The robots.txt file is written cleanly

Is my robots.txt written cleanly?

A crawler reads robots.txt one line at a time and skips whatever it cannot parse, silently. We look for directives a crawler does not recognise, lines with no colon, rules that come before any User-agent line, and a file longer than crawlers read. We show what we find and do not score it.

By Milk Moon Studio  Last updated

What it actually is

Robots.txt is a plain list of lines, each a directive and a value. The standard defines a small set of them, and every crawler reads the file line by line. A line it cannot make sense of is not an error it reports; it is simply ignored. We read the raw lines of the file and note the ones a crawler would have to skip or could misread: a directive that is spelled wrongly or that no mainstream crawler acts on, a line with no colon, rules written before any User-agent line so that no group owns them, and a file so long that the end is never read.

Why it matters

A machine only follows the rules it manages to read. A mistyped Disallow means the path you meant to close stays open, and a rule that sits above the first User-agent line applies to nobody. Because nothing complains, the file looks fine to the person who wrote it and does less than they believe. Search engines and AI crawlers share the same skipping behaviour, so one typo changes what every one of them is told.

What good looks like

Every line is a directive crawlers recognise, written with a colon, under the User-agent line it belongs to, and the whole file is short enough to be read in full. Comments, blank lines and Sitemap lines are all fine.

Being honest about it

Some of what we report is harmless: a directive only one search engine reads is still a line most crawlers skip, and we say so rather than judge whether you meant it. We read the file as served, so a plugin or a host that rewrites it shows up here as it really is. This check is shown in your report and does not change your score.

Where most sites go wrong

A directive is typed from memory and misspelled. A rule is pasted above the first User-agent line. A file is built by joining several sources and a line loses its colon. A generated file grows by a rule per page until the end of it is past where crawlers stop reading.

On the web

State of the Web · November 2026

In November 2026, 96% of the sites where it applies (83% of the sites we could read) passed the “The robots.txt file is written cleanly” check.

Pass 96% Worth a look 2% To fix 0% Couldn't judge 3%

Weebly sites pass it most often, at 100%. Vercel sites pass it least often, at 80%.

Not enough history yet.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

See every check on the web

Not sure where yours stands?

A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.

Scan your site free

Flagged for this one? We handle it. No pitch, just a plain-English chat.

Book a call