The AI crawler rules do not contradict each other

Do my AI crawler rules contradict each other?

We look for AI crawlers named in two groups that disagree, an Allow and a Disallow that cover the same path with the same length, and a group for one crawler that leaves out restrictions your general group has. The standard gives an answer in every case, so we show the answer it gives. We do not score it.

By Milk Moon Studio  Last updated

What it actually is

The standard says a crawler obeys the group or groups that name it and ignores the general group entirely. It merges every group that names the same crawler, lets the more specific rule win, and lets an Allow win a tie. Those rules are well defined and easy to forget. We read the groups for the AI crawlers we check and report where the result is probably not what the owner meant.

Why it matters

The common mistake is a short group for one AI crawler. Written to say something about that crawler, it silently stops the general group from applying to it, so the paths you closed for everyone are open to it. The reverse happens too: two groups for the same crawler disagree, and the crawler merges them and takes the more open reading. The file looks careful and says something different from what its author thinks.

What good looks like

Each AI crawler is covered by one clear set of rules. A group that names a crawler repeats every restriction the general group has that you want it to keep, and no Allow and Disallow rely on a tie to decide a path. Opening the whole site to a named crawler on purpose is fine and is not reported.

Being honest about it

No part of this is a mistake in itself; each is a legal file with a defined meaning, and an owner may want exactly that. We report the cases where the meaning is easy to miss, and the definition of which cases those are is ours, as no tool publishes one. We read the file as served and judge only the AI crawlers we check. This check is shown in your report and does not change your score.

Where most sites go wrong

A group is added to let one AI crawler in or keep it out, and the general restrictions are not copied into it. Two plugins each add a group for the same crawler. A rule is edited from Disallow to Allow by adding a line and leaving the old one.

On the web

State of the Web · November 2026

In November 2026, 95% of the sites where it applies (83% of the sites we could read) passed the “The AI crawler rules do not contradict each other” check.

Pass 95% Worth a look 3% To fix 0% Couldn't judge 3%

Weebly sites pass it most often, at 100%. BigCommerce sites pass it least often, at 5%.

Goes with

  • Sites that pass this are 2.5 times as likely to pass the “Open Graph tags for sharing” check too. 40% of the sites that pass the first check pass this one, against 17% of the sites that do not.

  • Sites that pass this are 2.5 times as likely to pass the “Twitter/X card tags” check too. 75% of the sites that pass the first check pass this one, against 28% of the sites that do not.

  • Sites that pass this are 1.5 times as likely to pass the “llms.txt is present and valid” check too. 29% of the sites that pass the first check pass this one, against 17% of the sites that do not.

Not enough history yet.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

See every check on the web

Not sure where yours stands?

A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.

Scan your site free

Flagged for this one? We handle it. No pitch, just a plain-English chat.

Book a call