Robots.txt lets crawlers reach the page and what it loads

Does robots.txt let crawlers reach my page?

We ask your robots.txt whether Googlebot, Bingbot and the AI crawlers we check may read this page, and fetch the scripts, stylesheets and images it loads from its own site. We report the answer for each. We never obey a Disallow ourselves, and this check is shown, not scored.

By Milk Moon Studio  Last updated

What it actually is

A crawler that follows robots.txt checks the address of every file before it fetches it. We run the same question over this page and over the page's own scripts, stylesheets and images, for the two search engines that render pages and for every AI crawler we check, and report which of them the file would turn away. Files on other sites are left out, because their own robots.txt decides.

Why it matters

A disallowed page cannot be read by a crawler that obeys the file. A disallowed script or stylesheet is quieter: a search engine that renders the page cannot load what it needs, and sees a broken or empty version. The same can happen to images the page depends on. None of this is visible to a person looking at the site, so it tends to go unnoticed until the page is missing from results.

What good looks like

Robots.txt lets the crawlers we check read the page and the files it loads from its own site. Keeping a crawler out of an admin area or a private folder is fine, as long as the pages and files that carry your content are not inside it.

Being honest about it

A disallowed page is sometimes exactly what the owner wants, and we say so by not scoring it. We only report what the file says: our own scanner reads a page regardless, because a scan is something a person asked for. We check a limited number of the page's files, so a long tail of resources is not covered. This check is shown in your report and does not change your score.

Where most sites go wrong

A rule written to hide a folder also catches the page template, or a rule for a whole assets folder catches the scripts the site needs to render. A staging rule that was never removed still disallows everything. A rule for one crawler is broader than the owner remembered.

On the web

State of the Web · November 2026

In November 2026, 85% of the sites where it applies (91% of the sites we could read) passed the “Robots.txt lets crawlers reach the page and what it loads” check.

Pass 85% Worth a look 12% To fix 0% Couldn't judge 3%

Weebly sites pass it most often, at 98%. Shopify sites pass it least often, at 23%.

Goes with

  • Sites that pass this are 3 times as likely to pass the “AI crawlers aren't blocked” check too. >99% of the sites that pass the first check pass this one, against 33% of the sites that do not.

Not enough history yet.

Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.

See every check on the web

Not sure where yours stands?

A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.

Scan your site free

Flagged for this one? We handle it. No pitch, just a plain-English chat.

Book a call