Are the robots instructions on my page clear?
Robots instructions should say one thing, in one place a crawler reads. We look for a nofollow, a robots tag that disagrees with the response header, a robots tag outside the head, the same tag repeated, and the rarely used directives that limit how a page is used. We show what we find and do not score it.
What it actually is
Beside the question of whether a page may be indexed, which has its own check, a page can carry other robots instructions. A nofollow says not to follow the links on the page. The same instruction can arrive twice, once in a tag and once in a response header, and the two can disagree. A tag can sit in the body, where a parser that stopped reading the head never sees it, or be repeated so that several readings are possible. A few rarely used directives, such as noimageindex and notranslate, limit how a page is used.
Why it matters
A crawler has to obey what it finds, and it has to find it. When two instructions disagree, search engines say they take the more restrictive one, which may not be what the owner meant. A nofollow on a page you wanted read tells a machine to ignore every link on it, so the pages behind those links are found less often. None of it changes how the page looks, so it goes unnoticed.
What good looks like
The robots instructions sit in the head, say one thing, and restrict only what you meant to restrict. If a response header carries an instruction too, it agrees with the tag. Most pages carry no robots instruction at all, which is also fine.
Being honest about it
Every finding here can be intended: a login page should not have its links followed, and a translated page may be better left as it is. We report them because the question this product asks is whether a machine can use the page, and each of these changes what a machine does with it. This check is shown in your report and does not change your score.
Where most sites go wrong
A template applies one page's nofollow to a whole section. A plugin adds its own robots tag and the theme adds another, so the page has two. A server rule adds a header for a whole folder that a page-level tag then contradicts. A tag pasted from an embed lands in the body.
On the web
State of the Web · November 2026In November 2026, 99% of the sites where it applies (91% of the sites we could read) passed the “Robots instructions are clear and in the right place” check.
SvelteKit sites pass it most often, at 100%. Contentful sites pass it least often, at 94%.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call