Is your site blocking ChatGPT, Claude and Perplexity?
Quite possibly, and probably without anyone deciding to. If your robots.txt blocks crawlers like GPTBot, ClaudeBot or PerplexityBot, those assistants cannot read your content — so when someone asks them about your industry, you are not in the answer.
What it actually is
AI assistants that gather content from the web do so with named crawlers, and those crawlers check your robots.txt first. That file is a guest list. It does not cover every fetch: some assistants also load a page at the moment a person asks about it, and their makers say that fetcher may skip robots.txt, because a person asked (OpenAI for ChatGPT-User, Perplexity for Perplexity-User, Meta for Meta-ExternalFetcher, Amazon for Amzn-User, and Google for its user-triggered fetchers). So robots.txt decides who may crawl you; it does not decide every request. The awkward part is that most site owners never wrote theirs: it came with the platform, or an SEO plugin added a blanket rule years ago, or somebody pasted in a "block bad bots" snippet from a forum. The list is enforced regardless of whether anyone meant it.
Why it matters
This is the check with the most direct commercial consequence of any we run. Search rankings are a competition you can lose slowly. Crawler access is binary: if a crawler cannot read the page, it has nothing of yours to draw on, no matter how good your content is. As more buying research starts inside ChatGPT and Perplexity rather than Google, a blanket block is the difference between being considered and being invisible.
What good looks like
A robots.txt that reaches a deliberate decision about AI crawlers rather than inheriting one. For most businesses that means the major answer-engine crawlers are welcome to your public content, and your genuinely private areas are the only things closed off. The important word is deliberate — a site that has decided to keep AI out for good reasons passes this on intent; a site that is blocking by accident is losing something it never chose to give up.
Being honest about it
Blocking AI crawlers is a legitimate choice, and we are not going to pretend otherwise. Publishers protecting paid archives, sites whose content is the product, anyone who has decided their words should not train a model — those are real positions and we do not mark them as failure for its own sake. What we flag is the gap between what your file says and what you think it says. Most blocks we find were nobody's decision.
Where most sites go wrong
The usual story is inheritance. A blanket rule blocks everything that is not a search engine, written before these crawlers existed and never revisited. Close behind: a rule that blocks one crawler by name while a newer one from the same company walks straight through, because the list was written to a snapshot of the internet that has since moved on. The third pattern is the most frustrating — the file is fine, but a firewall or bot-protection layer in front of the site refuses the crawler anyway, so the welcome mat is out and the door is still locked.
On the web
State of the Web · November 2026In November 2026, 92% of the sites we could read passed the “AI crawlers aren't blocked” check.
Webflow sites pass it most often, at 97%. WordPress sites pass it least often, at 86%.
On the sites scanned in both August 2026 and November 2026, it went from 77% to 75%.
The month-by-month line counts the sites scanned in each calendar month, so it can differ from the figure above, which uses each site’s latest scan from the last three months.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call