What is Content-Signal in robots.txt?
A newer addition to robots.txt that separates two questions people had been forced to answer together: who is allowed to read your site, and what they are allowed to do with what they read.
What it actually is
Traditional robots.txt is a door policy — a crawler is either in or out. Content-Signal is closer to terms of use posted beside the door. It lets you say that your content may be used to answer someone's question right now, but not to train a model; or that search indexing is welcome while other uses are not. Same visitor, different permissions for different purposes.
Why it matters
Without it you face an all-or-nothing choice, and most businesses get that choice wrong in one of two expensive directions. Block everything and you disappear from AI answers to protect content nobody was going to steal. Allow everything and you hand over material you would rather have licensed. Content-Signal is how you keep the visibility while setting terms on the rest — which for most service businesses is exactly the position they actually hold.
What good looks like
A robots.txt that states your position on the distinct uses — being found and cited in answers, versus being consumed as training material — rather than leaving it to be inferred from a crawler allow-list. The point is that your stance is written down somewhere a machine can read it, instead of living only in your head.
Being honest about it
This is listed under Agent & MCP readiness and does not move your score: stating your terms changes nothing about whether a machine can read the page. This is genuinely new and adoption is early — the operators who honour it are a minority today, and stating your terms changes nothing about how any machine reads the page. Treat it as a low-cost way of putting your position on the record rather than as enforcement. It is not a licence, it is not a contract, and anyone telling you it protects your content is overstating what it does.
Where most sites go wrong
The common failure is silence: nothing is stated, so every crawler applies its own default, and those defaults differ from company to company and change without notice. The subtler failure is contradiction — a site that blocks AI crawlers outright while separately declaring its content available as AI input. Whoever reads that has to guess which you meant, and their guess will not reliably be the generous one.
On the web
State of the Web · November 2026In November 2026, 4% of the sites we could read passed the “Content usage preferences declared” check.
Next.js sites pass it most often, at 11%. Webflow sites pass it least often, at 2%.
On the sites scanned in both October 2026 and November 2026, it went from 9% to 11%.
The month-by-month line counts the sites scanned in each calendar month, so it can differ from the figure above, which uses each site’s latest scan from the last three months.
Goes with
-
Sites that pass this are 11.5 times as likely to pass the “Clean content for agents (Markdown)” check too. 36% of the sites that pass the first check pass this one, against 3% of the sites that do not.
-
Sites that pass this are 6.5 times as likely to pass the “security.txt is published” check too. 25% of the sites that pass the first check pass this one, against 4% of the sites that do not.
-
Sites that pass this are 6 times as likely to pass the “AI bot rules are explicit and intentional” check too. 66% of the sites that pass the first check pass this one, against 11% of the sites that do not.
Not enough history yet.
Based on the sites Bot Appétit has scanned. Most of these sites come from lists we scan for research. Not a random sample of the web.
Not sure where yours stands?
A free scan checks this and everything else in about 20 seconds — no signup. Or if it's already flagged and you'd rather it were simply handled, we do that too.
Flagged for this one? We handle it. No pitch, just a plain-English chat.
Book a call