Should I allow or block AI crawlers?
Either is defensible — what is not defensible is not knowing. AI crawlers do several different jobs, and treating them as one category means whatever you decide about training data also silently decides whether you appear in AI answers at all.
What it actually is
The bots arriving at your site fall into roughly three groups: those gathering material to train models, those building an index so an assistant can find you later, and those fetching a page right now because a person asked a question about it. They are operated by the same companies and are often assumed to be interchangeable. They are not, and the consequences of refusing each are completely different.
Why it matters
Refusing training crawlers is a legitimate commercial position with a real argument behind it. Refusing the crawler that fetches your page because somebody asked an assistant about your business is simply refusing a customer. The two decisions get conflated constantly, and the second one is usually made by accident while making the first — the traffic disappears and nothing reports the loss.
What good looks like
A named position for each significant AI crawler that reflects what you actually decided, rather than a single sweeping rule applied to everything with "bot" in its name. Anything you have chosen to exclude is excluded deliberately, and anything you want quoting you can reach the pages you want quoted.
Being honest about it
We have no view on which way you should decide and we are not going to pretend there is a correct answer — publishers with valuable archives and businesses wanting to be found in assistants are in genuinely different positions. What we do check is coherence. The failure we care about is a site that meant to protect its content and has instead made itself invisible, or one that never made a decision and does not realise a default was chosen for it.
Where most sites go wrong
The most common failure is the blanket rule adopted during the first wave of concern about AI training, which also excludes the crawlers that would have brought referrals. The second is drift: rules added by different people over two years that now contradict each other, with the effective outcome decided by ordering rather than by anyone's intent. Both look deliberate from the outside and neither is.
On the web
State of the Web · November 2026In November 2026, 13% of the sites we could read passed the “AI bot rules are explicit and intentional” check.
Next.js sites pass it most often, at 27%. Webflow sites pass it least often, at 9%.
On the sites scanned in both October 2026 and November 2026, it went from 24% to 23%.
The month-by-month line counts the sites scanned in each calendar month, so it can differ from the figure above, which uses each site’s latest scan from the last three months.
Goes with
-
Sites that pass this are 13 times as likely to pass the “Content usage preferences declared” check too. 19% of the sites that pass the first check pass this one, against 1% of the sites that do not.
-
Sites that pass this are 7.5 times as likely to pass the “Resource-discovery Link headers” check too. 6% of the sites that pass the first check pass this one, against 1% of the sites that do not.
-
Sites that pass this are 4.5 times as likely to pass the “security.txt is published” check too. 14% of the sites that pass the first check pass this one, against 3% of the sites that do not.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call