Connect any MCP client to botappetit.ai/mcp
Any MCP client can now connect to botappetit.ai/mcp and scan a page, read a stored report, follow a page's history and look up what a check means. The same tools are open over a public API, described in an OpenAPI document. Every answer links back to the full report. Single-page scans stay free and need no account. A whole-site scan still needs an account, so the connection tells the assistant exactly what the person has to do. No check or score changes.
Link to this entry
Four new checks, shown in your report and not scored
Your report now shows four more things a machine cares about. How much it costs an AI to read your page, whether the page hides text written for AI models, whether the dates in your sitemap tell a crawler what changed, and whether the links in your llms.txt open. They sit with the measured-but-not-scored checks, so they appear in your report without changing your score. Where we report hidden text we say how it is hidden and where to look, and never repeat the text itself.
Link to this entry
Fewer false passes on sites that answer missing files with a styled error page
Some sites answer a request for a file they do not have with a styled "page not found" page instead of a not-found status. The scanner now recognises more of those pages, so it no longer reports such a file as present. Separately, a page whose markup carries an old commented-out script could be read as nearly empty; the scanner now reads the text after it correctly. Scores could move for affected sites; on our reference set of real pages none did.
Link to this entry
Embed a State of the Web figure on your own site
Any headline figure on the State of the Web page can now be embedded on another site. Open the Embed or share this figure control, choose dark or light, and copy the code. The embedded figure stretches to fit the space you give it, up to 720 pixels wide, and links back to the full page. It is hidden from search engines, so it never competes with the page itself. A figure we do not yet have enough sites for cannot be embedded. No check or score changes.
Link to this entry
Share a State of the Web figure, and read past editions
Each headline figure on the State of the Web page now has an Embed or share this figure control. Open it to see the card for that figure and copy a dated sentence with a link back to the page. The page's link preview is a new card too. A figure we do not yet have enough sites for gets no card. The AI access section now sets out the sites that allow AI crawlers in robots.txt and still turned our scanner away as a card of its own. The page for a dated edition now lists the past editions. No check or score changes.
Link to this entry
A redesigned State of the Web
The State of the Web page is easier to move around. A contents list stays with you as you scroll, and the view switcher and headline cards are clearer. Every check now sits in one table you can search, filter by category and sort. Change over time is one chart: pick up to four lines and see how they moved, with the days we changed how we judge marked on it. Each platform's page shows whether that platform sits above or below the average site, and every check page has an On the web section. The page also shows how the sites we re-scan are doing, how fast the web is, how far sites climb the readiness ladder, where they differ by platform and hosting, which checks tend to go together, how each category fares and which emerging standards sites adopt. Each is drawn as a chart with the same figures as a table. Section headings have a Copy link button. The homepage sample report now shows our own latest real scan. No check or score changes.
Link to this entry
A fuller homepage, a Features page and a Scan menu
The homepage now explains the whole product: what Bot Appétit is, how a scan works, a sample report of our own site, why you can trust the score, who it is for, what a site scan adds, a live State of the Web preview and what a scanner cannot tell you. A new section shows everything you can do, with a picture of each and a note on whether it needs an account. A new Features page lists every feature in one place. The header now groups the two ways to scan under one Scan menu and links to Features, and the footer links there too. A new account's empty Your sites page says what the account does, and the monthly re-scan switches look tidier, with clearer saved and error messages. The FAQ also answers whether we re-scan your site for you. No check or score changes.
Link to this entry
Report a bug and request a feature on GitHub
Report a bug and Request a feature now open their own pages instead of an email. What you send is posted as a public issue on GitHub, so you can follow it, and you do not need a GitHub account. If you do have one, each page links straight to the matching GitHub form with what you have typed already filled in. A reply email is optional and stays private. No check or score changes.
Link to this entry
Titles, headings and redirects are read the way a browser reads them
Special characters written as named entities in a page title, heading or description, such as the middle dot or curly quotes, are now read as the character they stand for, so title lengths are measured on what visitors see. A heading that contains another heading now keeps all of its own text. A page-refresh redirect placed inside a noscript block is now noticed, because a reader that runs no scripts follows it. Scores could move in either direction for pages that use these.
Link to this entry
Organization markup is recognised for every kind of organization
Organization markup is now recognised for every kind of organization schema.org defines, such as schools, salons and bands. An empty profile-links or logo field no longer counts as filled in. A FAQ attached to a business is now found wherever the markup places it. Scores could move in either direction for pages that use these.
Link to this entry
State of the Web counts each feature once and drops empty sections
Checks that are one feature published as several files now count once in the platform comparison, so one platform default no longer fills the whole ranking. What goes together leaves out pairs that mostly come from one platform or one research list, and counts a site and its staging copy as one site. The Scanned by visitors view no longer draws sections that have no figures, words its captions for that view, and says the group is small. Nothing about how pages are scored changes.
Link to this entry
State of the Web stops drawing one-bar charts and explains newer checks
When a platform has too few sites to split by score, How sites land now says so in a sentence instead of drawing a single bar at 100%. The published figures no longer carry the verdict names, only the score ranges. The Every check section also notes that newer checks only cover sites scanned since the check was added. Nothing about how pages are scored changes.
Link to this entry
State of the Web shows score ranges over time, and says when checks tie
The Change over time section now follows the same plain ten-point score ranges as How sites land, one line per range, in place of the verdict names. A range appears for a month only when enough sites score in it. The Visitors versus research lists table uses the ranges too. On each platform page the Gap column reads in points rather than a percent sign, and when several checks pass equally rarely the page says how many instead of naming a few of them. Nothing about how pages are scored changes.
Link to this entry
The State of the Web only compares pairs of checks that enough sites pass
A "times as likely" comparison used to be able to rest on one or two sites. A pair of checks now appears only when enough sites pass the second check both among the sites that pass the first and among those that do not, and each pair shows both percentages beside the comparison. One check opens at most two rows on the list, checks that repeat each other (such as a Markdown version being served and being linked) are no longer paired, and the OAuth and OpenID discovery checks count as one. The same rules apply to the "Goes with" block on each check page, which is hidden on checks almost every site passes. Nothing about how pages are scored changes.
Link to this entry
State of the Web shows fewer empty rows and says what each figure is a share of
Tables on the State of the Web no longer draw a row or a column where every figure is "not enough sites yet", and a check with a single month of figures is linked to the Every check table instead of being listed twice. Platform pages now say they are shares of the sites we could read, so their score ranges add up to 100%, and the platform buttons on the main page open each platform's own page. Sections that cover all sites say so when you look at one platform. The two Organization figures say what each one counts. No figure is removed, and nothing about how pages are scored changes.
Link to this entry
State of the Web share cards show the latest figures
The picture shown when you share the State of the Web, and the cards you can embed from it, could keep showing the figures from the first time they were drawn. Each card link now belongs to one update of the figures, so a new update gets a new picture. No figure changes, and nothing about how pages are scored changes.
Link to this entry
Every State of the Web check figure now says what it is a share of
A check that has nothing to judge on many pages, such as FAQ schema on a page with no questions, is now shown with the share of sites it applies to, and its headline sentences say "of the sites where it applies". The Method says check figures count only the sites we could read. The check "Automated visitors can reach your pages" no longer shows a share: it passes whenever our scanner gets in, so the figure that matters is how many sites turned the scanner away. "Disallows some AI crawlers" now includes the sites that disallow the major ones. Nothing about how pages are scored changes.
Link to this entry
State of the Web charts show each figure once, and "How sites land" counts scores
Each chart on the State of the Web used to be followed by a table repeating the same numbers. The numbers now sit with the bars, and the table is one click away under "Show as a table". "How sites land" now counts the score at the top of each report in plain ten-point ranges, from 91 to 100 down to 0 to 10, with the sites our scanner could not read as their own row. Nothing about how pages are scored changes.
Link to this entry
The State of the Web only states a change over time for the same sites
The sites we scan change from month to month, so a rise in the share of all sites can be a different set of sites rather than the web changing. "Change over time" and each check's "On the web" block now state a change since the series began only for the sites scanned in both months, and say so. Where too few sites were scanned in both, they give the current month alone. The check pages also explain their two time windows and no longer name a best and worst platform that are tied. The note under each figure now says most of these sites come from lists we scan for research. Nothing about how pages are scored changes.
Link to this entry
When we can't reach a page, we record why
A scan that couldn't read a page used to say only that it was unreachable. It now keeps the reason we were given, such as a dropped connection or a read that stopped halfway, so a page that fails once can be looked into instead of guessed at.
Link to this entry
Site scans no longer test a "#" or "back to top" link as a link
Site scans now treat a "#" or "back to top" link the way single-page scans do: a link to the page itself no longer takes up one of the links we test on each page. The link checks in a site scan only judge links that go somewhere.
Link to this entry
Four lists on the State of the Web say more and repeat less
The hosting list now counts only networks that sit in front of a site, not script libraries a page loads from. The "what goes together" list shows each pair once, with its ratio as a number. Platform pages lead with what sets the platform apart from all sites, and a view with too few sites no longer gets its own button or page. The figures that remain are unchanged, and nothing about how pages are scored changes.
Link to this entry
The State of the Web "Change over time" section is much shorter
Most of the lines in "Change over time" only have one month of figures so far, and each took a whole block of the page. Those now sit together in one table, and each becomes a month-by-month line once a second month of figures arrives. Lines with two or more months look the same as before. No figure changes, and nothing about how pages are scored changes.
Link to this entry
Placeholder links no longer count as internal links
A menu item or button whose link goes nowhere (a bare #) used to be counted as a link to another page on your site, so a page could look well linked when it had almost no real internal links. Those placeholders are now ignored, so the internal-linking result reflects the links a crawler can actually follow.
Link to this entry
Pages with search boxes and lazy-loaded images are read in full
A page that marked a search box or an image with structured-data attributes could stop the scanner mid-page, so most of its checks were skipped and it looked emptier than it was. The scanner now reads those pages to the end and runs every check.
Link to this entry
Link checks no longer test a link to the page itself
A "#" or "back to top" link used to take up one of the links we test on each page, and on a page made only of such links it made the redirect check appear to pass. They are now left out, so the link checks only judge links that go somewhere.
Link to this entry
The State of the Web, and how common each check is
There is now a public page showing what share of the sites we have scanned pass each check, by category and over time, with the same figures as a download. Every check page also shows how common that check is on the web. All figures are percentages that refresh every hour, and a figure appears only when enough different sites sit behind it. Nothing about how pages are scored changes, so no score moves because of this.
Link to this entry
Your sites are now re-scanned every month
Every site in your account is now scanned again 30 days after its last finished site scan, and we email you the result with the score from last time. A site scan you start yourself restarts the 30 days. Two switches under Monthly re-scans on Your sites turn the re-scans or the emails off, and removing a site or deleting your account stops them. If you have not signed in for about three months we email you to say the re-scans will pause, and pause them a week later if you still have not; signing in starts them again. The scoring is unchanged: a re-scan is the same site scan, so no score moves because of this.
Link to this entry
The sitemap your robots.txt points to is now checked
If your robots.txt names a sitemap, we now ask for that address and look at the answer. A sitemap that answers normally passes, and one that answers "not found" or "gone" now fails, because that is the address crawlers follow. When the address will not answer us at all, because of a firewall, a rate limit, a server error or a timeout, we say we could not judge it instead of passing it. That can lower a score slightly, because a pass we have not actually checked no longer counts. It is never marked as a failure.
Link to this entry
Structured data about a page's main subject is now read
Structured data that describes the page's main subject, such as an organisation or a person named in the page's "about" or "main entity", is now read when we check for an organisation, an author and typed schema. Pages that describe themselves this way were reported as missing it.
Link to this entry
A site whose secure version answers slowly is no longer reported as not using https
When a site's secure version was too slow to answer during our scan, we used to read its plain http page and report that the site is not served over https. We now say we could not judge it, and why. The check on what a page loads over insecure connections says the same in that case, because we only saw the http page. Sites that genuinely refuse a secure connection are reported as before.
Link to this entry
A slow secure version no longer marks a site down for its canonical or HSTS
In the same situation we also warned that a site's https canonical "points somewhere else", only because we had landed on the http page, and counted a missing HSTS header against it, although browsers ignore HSTS on http. We now compare the canonical without the web address's scheme and say plainly that we could not judge HSTS, while the other security headers are checked as usual. Sites that genuinely refuse a secure connection are checked as before.
Link to this entry
Analytics is found wherever it sits on the page, and more tools are recognised
The analytics check now reads all of a page's inline code, so a tracking tag that loads late, such as one set up by a cookie-consent tool or a page builder, is no longer missed. Self-hosted Umami, Burst Statistics and Independent Analytics are now recognised, and the word "Mixpanel" appearing in a link or a list of tags no longer counts as Mixpanel being installed. This check is advice and is not part of the score.
Link to this entry
The www and non-www check no longer skips when the other name does not exist
When your site answers on only one of www and the bare domain, and the other name simply does not exist or has an invalid certificate, the check now counts that as one canonical host instead of saying it could not tell. Scans where the other name refused or timed out still say so. Scores can move on the sites this affects.
Link to this entry
Analytics look-alikes no longer count, and page titles read the way a browser reads them
The analytics check no longer counts look-alikes: something that only resembles an analytics tag, such as a contact form's spam protection, is not analytics, so a site without real analytics now gets the warning it should. And when a page carries a second, embedded HTML document with its own title, we now use the page's real title instead of joining the two, so title-length results match what visitors and search engines see. Scores can move on the pages this affects.
Link to this entry
Older reports now add up, and slow sites say so
On a scan made before we re-sorted a few checks, each section's tally now matches the checks listed under it, and nothing is left out. An older site scan now measures what it judged against what that scan could run, not against today's list. A scan that ended because a site could not be reached now names the site and says why. And a site scan that could not see the whole site, because the site was too slow while we looked for its pages or because nearly every page we sampled led to the same one, now says so above the score. No score changes.
Link to this entry
Short brand names are compared, and checks we could not run say why
A page whose title and main heading are short brand names, such as an acronym, is now compared instead of skipped, and a page made only of very common words says that is why it was not compared. When a site turned our scanner away, each check we could not run now says so instead of a generic line. The www and non-www check is more patient with a slow other host, and when it still cannot tell, it names the host and what happened. Scores can move a little on pages these touch.
Link to this entry
Five checks stopped misreading a few kinds of page
A search-console verification we could not look up is now left out of the score rather than counted as missing. The secure-redirect check looks at the address your site actually lands on, not only the one we were given. A heading like "Any questions?" above a contact form is no longer taken for an FAQ. Webflow Ecommerce stores are now recognised as stores by the shop-related checks. The broken-links result says how many links it tested, so you can tell a sample from the whole page. Scores can move a little on pages these touch.
Link to this entry
The privacy page now covers one visitor across our sites
Usage counts now treat a visit to www.milkmoonstudio.com and a scan here as one visit, using a cookie on milkmoonstudio.com. For people who sign in, or book a call through the scanner, our analytics also receive the email address and the Google Analytics and HubSpot visitor IDs from the browser, and the IDs are kept with the account. The privacy page says exactly this.
Link to this entry
Usage counts now reach PostHog through our own domain
Your browser now sends our usage counts to PostHog through an address on our own domain that PostHog runs. What PostHog receives is unchanged, and the privacy page says so.
Link to this entry
The Book a call box names what to fix first
When your page has things to fix, the Book a call box under your results now names the first one or two of the checks that matter most, in plain words, next to how many need fixing. No scoring changed, and the order of the checks is not published.
Link to this entry
A quick way to ask for a feature
There is now a Request a feature link next to Report a bug, in the footer and at the end of the Learn menu. It opens an email to us with a few prompts to fill in.
Link to this entry
New checks, and fewer signals only a person can see
The scan now asks whether engines are allowed to quote your page, whether your buttons, links and form fields have names a program can read, whether the page arrives quickly enough for an AI assistant to wait for it, whether the HTML is small enough for Google's crawler to read all of it, and whether your links are real links a crawler can follow. Structured data published as microdata or RDFa now gets credit, the same as JSON-LD, and a new check looks for the facts in your markup in the words of the page, because some AI fetchers read only the text. Another new check looks at whether your claims come with figures, a quoted source or links to where they come from; it is shown and not scored, as is a look at whether your section headings are phrased as the questions they answer. Some judgements are fairer: more than one H1 is no longer marked down, the meta description is judged on being there, a page with no headings is no longer reported as several separate problems. Finally, the viewport tag, security headers, favicon, link hygiene, Content-Signal and Link headers still run and still appear in your report, but they no longer count towards your score: none of them changes whether a machine can read your page. Because of all this, scores and verdicts can differ from earlier scans of the same page.
Link to this entry
More ways AI agents can learn about your site
The Advanced course now also looks for an agent card, WebMCP tools declared in the page, AI usage preferences, licensing and data-mining reservations, a linked Markdown version of the page, and whether your content signals agree with each other. None of them counts toward your score, and a missing one is normal. The wording on the llms.txt, Link header and AI crawler explanations is also corrected: crawlers check robots.txt, while a fetcher that loads a page because a person asked may not.
Link to this entry
The Book a call box now starts from your result
The Book a call box under your results used to say the same thing to everyone. It now says what we found on your page: that we couldn't read it, how many checks need fixing, or that nothing urgent turned up. If your site is on Webflow and has things to fix, it says so. The scan itself is as free and complete as before, whether you book or not.
Link to this entry
Clearer site scans when something goes wrong
If a site scan can't start, we now email you to say so instead of leaving you waiting. When PageSpeed can't measure your homepage during a site scan, the report says whether it timed out or couldn't get in. A partly finished site scan no longer says we were refused and that we scanned your pages in the same breath. A scan's permalink now says when it had to cut your structured data short instead of hiding it, and the brand name keeps its accent in headings.
Link to this entry
We now see where the tool confuses people
Our usage counts now include what gets clicked and tapped, how far down a page people scroll, how quickly pages load, and errors in our own page code, so we can find and fix the confusing parts faster. We still never record what you type or replay your visit, and the privacy page lists exactly what is collected.
Link to this entry
Our privacy page now names the tool we count usage with
We now count how Bot Appétit gets used (scans run, site scans started and finished, sign-ups) with PostHog, on its European cloud, so we can see which parts earn their keep. The privacy page names it and says exactly what it receives and what it doesn't. It sets no cookie of ours and records no screens.
Link to this entry
Compare any two scans
Pick any two scans of a page, or two of your site scans, and see them side by side: both scores, every category's change and every check that moved. If our scoring changed in between, the change still shows, with a note that our scoring changed and a link to what we changed. If the pinned pages of a site changed between two site scans, we say their scores aren't compared instead of printing a number; each report still stands on its own.
Link to this entry
Every check we run now has its own page
Until now only some of the checks had a page of their own. Every one does today, from the security basics to the agent and MCP readiness checks. Each page says what the check looks at and why a machine reading your page cares. The checks on your report link straight to their page, including the ones that sit outside your score, which now explain why they do. Nothing about how any check decides has changed.
Link to this entry
Help guides, the checklist and the answers now live on the scanner
Short guides at /help/ now answer the how-do-I questions in one place, from scanning a page and reading your report to scanning a whole site, sharing a scan, letting our scanner in and recovering from a scan that did not work. The Webflow SEO and AEO checklist has moved here from its old address, which now forwards to the new one. How it works, the FAQ and a page about who makes Bot Appétit have their own pages. The site also publishes llms.txt and llms-full.txt, built from the same pages, so an AI assistant can read all of this too.
Link to this entry
A site scan now names the pages your robots.txt disallows for us
If your robots.txt disallows our scanner from some of the pages we sampled, the site scan report now says which pages and which rule matched. We still scan them and they still count; the report simply tells you that a machine honouring your rules would not have read them. This is a reporting change and it moves no score by itself, but our scoring version advanced with it, so a comparison across this date carries the usual note.
Link to this entry
Deleting your account now erases your email, sites, pinned pages and site scans
Our privacy page promised that deleting your account removes your details, and the deletion did not fully do that. It does now. When you delete your account, your email address, your sites, your pinned pages and your site scans are erased, along with the contact record we keep from sign-up. The results of page scans stay, because those belong to the page and not to you, and anyone who knows the address can scan it themselves.
Link to this entry
A directory of AI crawlers, with who runs each one and what it is for
/crawlers/ lists the AI crawlers our checks look for, with the company behind each one, what it is used for, what its maker says about robots.txt and the day we last read that page. You can filter by purpose and by robots.txt stance. It reads the same list the scanner uses, so the two cannot drift apart. The directory changes no score, but our scoring version advanced with it, so a comparison across this date carries the usual note.
Link to this entry
A glossary, a page for each course of checks, and a hub for performance
/checks/ is now the front door to every check, with a hub for each course listing its checks and the words they use. A glossary at /glossary/ explains the terms in plain English, with a page each for the ones that deserve it, such as AEO, Web Bot Auth and schema.org. Performance has its own hub, written to say what we measure and nothing about where a page will rank.
Link to this entry
Your sites and each site's page were rebuilt around the history
Your sites is now a table with each site's score beside its history chart, and a site's own page opens with its score, what changed since the last scan, the shape of the site against its pages and every earlier scan. "What changed" shows the movement as bars, and a score that moved because our scoring changed says so. Sharing a scan now tells you what happened, and the person you share with never meets a sign-in wall. A homepage performance reading that is still being measured says "in progress" instead of "not recorded".
Link to this entry
Every score change is shown, and every change in our scoring links here
A page's report now draws its score history as a line, with the scan you are looking at beside the one before it. Every change in score is shown, small ones included, in the same chip on the report, in Your sites and on a site's page. When two scans were scored by different versions of our scoring, the chip still shows the number, says "our scoring changed" and links to this page. A report with no score now says "No score" and why, on every kind of report, and each error screen says whose side the failure was on.
Link to this entry
A new look, and plainer words for what we found
The header, footer and homepage were redesigned, and the scan box finally has a visible label. On the report, the verdict now leads on a band of its own with what to do next, and there is one score ring instead of two that looked alike. Results use the same four words everywhere, Pass, Worth a look, To fix and Couldn't judge, each with a symbol as well as a colour, so a status never rests on colour alone. "Rescan this page" rescans that one page. Everything that moves respects your reduced-motion setting, and the grey used for secondary text now passes AA contrast on our background.
Link to this entry
Pin the pages that matter and they join every site scan
You can now pin pages to a site you track, from the site's page. Pinned pages are added to every later site scan and count towards the site score like any sampled page, so a checkout or a pricing page is never missed by the sample. The report marks the pinned pages, and history says when a score moved because the pins changed rather than the site. The signed-in scan form also lets you choose between a single page and a whole site.
Link to this entry
A site scan now measures the performance of your homepage
A site scan is meant to measure your homepage's performance, and it was not doing so. It does now: when the scan finishes, the homepage's performance reading is added to its page scan and shown on the site's overview. A finished site scan also no longer waits for that reading before it tells you it is done. Other pages can still have a performance scan added from their own result, which runs a fresh scan of that page.
Link to this entry
A page withheld for legal reasons is no longer called bot protection
When a page answers that it is unavailable for legal reasons, such as a regional block or a court order, our report used to say bot protection had refused us and offered robots.txt lines that could not help. It now says the page is withheld for legal reasons, offers nothing to allow, and says the same applies to AI crawlers.
Link to this entry
A paywall or a rate limit is no longer called a firewall
When a site scan stopped early, the summary said your firewall had begun blocking us, whatever had actually happened. A rate limit is now described as a rate limit, a page that asks you to sign in as a sign-in, and a paywall as a paywall, in the summary, the status and the page rows alike. Before, one report could describe the same event three different ways.
Link to this entry
A "Robot or human?" screen is no longer scored as your site
Some sites answer our scanner with a bot-check screen that looks like an ordinary page at a glance, with a title and a heading and nothing else. We were scoring that screen as if it were the site. It is now recognised as a wall, so the report says we were turned away and gives no score built from the wall.
Link to this entry
A shared scan now unfolds into a card with the score
Paste the link to a scan into a chat or a social post and it now previews as a card showing the score, the verdict, the page and the date. A scan with no score shows none, rather than a number we cannot stand behind. The card and the page title come from the same source, so they cannot disagree.
Link to this entry
Pages that build themselves in the browser are now read the way a browser reads them
Some pages send almost nothing until JavaScript runs, so a crawler that reads only the first response sees an empty shell. A new check notices that, loads the page in a real browser and reports how much of your content only appears after scripts run. Where the browser cannot load the page the check says it could not judge rather than guessing. It works the same inside a site scan.
Link to this entry
Site scans now use all the pages they are allowed to read
A site scan has a budget of pages it may read. Many scans stopped before spending it, even when the site had more pages worth sampling, so they covered less of the site than they could have. They now keep going until the budget is spent, which means a site scan can read more of your site than it used to and its score can differ from an earlier scan of the same site.
Link to this entry
Turned away? The report now tells you how to let us in
Some sites answer our scanner with a wall rather than a page — a firewall rule, bot protection, or a challenge screen — and until now a single-page scan just told you it happened. It now tells you what to do about it, and what to do depends on what actually stopped us: a refusal gets the two lines that allow our user agent and a note that an edge block happens before robots.txt is ever read; a rate limit gets "wait a few minutes", because we were slowed rather than refused; a page behind a login gets neither, because there is nothing to allow and that is itself the finding; and an address with nothing published at it is not treated as a block at all. Our bot page now carries the detail too, including the one case where a Cloudflare site cannot make an exception at all. Letting us in only lets us in — it changes nothing else about the report, and every finding in it still stands.
Link to this entry
A template that scored two ways now says so
A site scan reads a sample of your pages rather than all of them, and groups pages built the same way so one finding can speak for the rest. Until today the report left a single page's score standing for its whole template — and when we measured that, it turned out to be too generous: on almost half the templates where we had read more than one page, those pages did not score the same, and on some sites the number you saw depended on which page we happened to pick. So the breakdown of your site now shows what the pages we actually read scored — the range, and where there are enough of them, the middle. Where the pages agreed, it says they agreed. Where your site has more templates than the breakdown can list, it says so and shows no ranges at all, rather than ones we cannot stand behind. Nothing about how we score changed, and no score moved.
Link to this entry
"AI access" now means access, and what you tell AI has its own place
One number on your report was answering two different questions at once. "AI access" mixed whether AI can actually reach and read your pages — which is either true or it is not — with three optional extras you can declare for AI clients once they are in. A site whose access was completely fine could sit at three-quarters simply for not declaring the optional things, and a site that was genuinely turning automated visitors away could be propped up by declaring them. Neither reading was fair. From today they are two courses. "AI access" reports only whether AI can get in, so a good number there now means what it says. "AI signals" reports the things you tell AI clients — your crawler rules, your content-usage preferences, and the headers that point a machine at your sitemap. Every check we run is the same as it was yesterday, and every one of them still counts exactly as much toward your overall score, so that number has not moved. What moved is the two smaller numbers underneath it: most sites will see "AI access" rise, and sites that really were blocking automated visitors will see it fall to something closer to the truth. The advisory under your verdict is unchanged, and deliberately so — a missing optional extra is not a fault, so we do not call it one.
Link to this entry
We look in more than one place before telling you something is missing
Three checks were reaching a verdict after looking in a single place, and all three could be wrong about a site that was doing the right thing. If you serve a clean, plain-text version of a page for AI assistants, we used to notice only if you offered it one particular way — so sites doing exactly what we recommend were told they were not. We now look both ways before saying it is absent. If your site tells us where your sitemap lives, we now follow that even when your address quietly redirects from the bare domain to the www one, and we still fall back to the usual location when what you published there does not lead anywhere. Between them those two things were the main reason a multi-page scan sometimes ended up following links from your homepage instead — a much shallower view of a site, because your navigation is not your content. When that does still happen we now say so on the report, and tell you that you can paste your sitemap's address straight into the scan box and run it again. Separately, when we judge whether AI crawlers can reach you, we now give less weight to turning away the fetchers whose own makers publish that our rule may not bind them — being blocked still gets said plainly, it simply counts for less toward "you cannot be cited" than blocking a crawler that genuinely obeys. In practice that changed no site's result today; it will matter for the sites where it would have been the wrong call.
Link to this entry
Your site score now reflects how much of your site is actually right
A site score is built from the pages we sampled, and until today it was harsher than any of them. A check that was right on forty-nine pages and wrong on one counted the same as one that was wrong on forty-nine and right on one — so almost every site score sat below the score of every page beneath it, and some sites read a whole grade worse than the pages they were made of. That is now fixed: getting something right on most of your pages counts as getting it mostly right. The wording on each finding has not changed, and neither has the order of what to fix first — you will still see "passes on 41 of 50 pages" and know exactly where you stand. Alongside it, the files that belong to your whole site rather than to one page — your robots.txt, your sitemap and the rest — are no longer judged on an occasional request of ours that failed to land. We ask for them once per page we scan, and if a handful of those requests time out while the rest come back fine, we now go with what your site told us most of the time instead of with the worst answer we got. Site scores move up as a result, most of them by several points and some by considerably more. Individual page reports are untouched.
Link to this entry
Your score now counts only what decides whether a machine can read your page
Bot Appétit exists to answer one question: can a machine find, fetch, parse, trust and cite your page? Two changes bring the score closer to answering only that. Six checks that were never about it — things like the length of your title, whether an analytics tag is installed, and how well your images are compressed — no longer move your number. They are all worth getting right, so we still run them and still show you what we found; you will now find them grouped under "Good practice", apart from the checks that count. And in a site scan, pages you have explicitly told search engines to ignore are set aside rather than scored, because judging your site on them was measuring which pages we happened to sample rather than your site. The report says how many were set aside, and if every page we sampled turns out to be marked that way we keep them — at that point it is a fact about the site rather than an accident of our sampling. Most scores move by a point or not at all, and more go up than down. Individual page reports are unchanged: scan any URL directly and you still get its full result.
Link to this entry
The report now says what counts towards your score, and keeps our own testing out of your history
Performance and the agent and MCP readiness checks are shown on your report but do not feed the headline, and until now they looked exactly like the categories that do, so fixing one and watching the score stay put made no sense. They are now labelled as measured but not counted. On a page's result, adding a performance reading runs a fresh scan of that page and adds a new line to its history, never a measurement attached to an older result. And the scans we run ourselves for testing no longer pad the public history of a page, so "scanned N times" counts real scans.
Link to this entry
Every scan now records which version of our scoring produced it
Your report history can now tell the difference between "your site changed" and "we changed how we measure". Older scans have been labelled retrospectively where we could establish it with certainty, and left unlabelled where we could not — we would rather say "we cannot tell" than guess.
Link to this entry
A category we could not measure no longer shows as a zero
When a page could not be read completely, some groups of checks never ran. Those groups were being shown as a red zero, which read as "you failed this" when the truth was "we could not look". They now show a dash, and say so. Overall scores are unaffected — the checks were already excluded from the maths.
Link to this entry
Rebalanced one check that was counting for more than it proved
One technical check almost never found a real problem, so in practice every site was treated the same by it. We adjusted how much it counts, which moved most scores by a single point or not at all, and left a little more of the result resting on the answer-engine signals this tool exists to measure.
Link to this entry
Share a site scan with an unlisted link
A site scan belongs to the account that ran it, and the list of pages in it stays private so a pasted list of unlisted addresses is never discoverable. The owner can now share the scan with a link. Whoever opens it sees the whole report without signing in, the link is not listed anywhere, and the owner can stop sharing at any time. The results of any single page stay public to anyone who knows its address, as they always were.
Link to this entry
A missing llms.txt is reported as a fail again
This had been softened to a warning, which contradicted our published standard. We do not lower a bar because most sites do not clear it.
Link to this entry
Site scans no longer count the same page more than once
Several links can lead to one page. When they did, that page could be counted several times in a site's score. It is now counted once, however many URLs reach it.
Link to this entry
A site that blocks our scanner gets no score at all
If your firewall turns us away, we can only see a handful of things — not enough to judge a site on. Rather than publishing a number built from fragments, we now publish none, and explain what happened and how to let us in.
Link to this entry
Accounts and site scans arrive
You can now create an account and scan a whole site rather than a single page. A site scan reads a sample of your pages in the background, so it can take a few minutes, and the results are tied to your account. Your sites lists every site you have scanned, with a page for each site that keeps its history. Scanning a single page still needs no account.
Link to this entry
Every scan gets its own link, and every page gets a history
Each scan now has a link of its own that opens the report exactly as it was, without scanning again. A report also shows the history of the page it is about, whoever scanned it and whenever, with the change in score and a link into each earlier report. The link previews as a card for that scan, and the performance shown on an old report is the performance recorded at the time, not today's.
Link to this entry
Our scanner now says who it is, and proves it
/bot/ documents our scanner for the people who see it in their logs: what it requests, how often and why. Our requests are now cryptographically signed with Web Bot Auth, the same way the big AI crawlers are, so a site's bot protection can verify them instead of guessing. Sites that turned us away on every address because they could not tell us apart from an impostor now have a way to recognise us.
Link to this entry
A site that blocks AI crawlers can no longer look healthy
A site that explicitly told the major AI crawlers to stay out could still score a top verdict, because its pages were well structured, which is exactly what a crawler never gets to see. Optimisation behind a locked door is not a degree of readiness, it is the absence of it, so a block on AI crawlers now holds the verdict down however well the rest of the page is built.
Link to this entry
The AI crawler list was refreshed, and page weight is measured properly
The AI crawlers we look for now include the newer search and assistant crawlers from Anthropic, Perplexity, Mistral and DuckDuckGo, each confirmed against its maker's own documentation. Page weight and render-blocking resources are now taken from PageSpeed Insights and Lighthouse rather than estimated. The MCP server card check accepts all three ways the specification has described publishing one, and a check for agent registration, auth.md, was added earlier in the week. Checks that nothing displayed or measured were removed from the list.
Link to this entry
The verdict now follows how well AI can read you, not just the total
A site could be strong at the things that matter least to this tool and score a green verdict while being weak at the one it exists to measure. The number stays honest, but the verdict is a promise, so it can no longer be better than your AI-readiness. When one course is much weaker than the rest, the report also says so under the verdict, so a good total cannot hide it.
Link to this entry
Four new checks, and a caching check
We added a check for a favicon, one for a meta refresh used as a redirect, one for a published security.txt contact file and one for breadcrumb structured data on inner pages, which is not expected on a homepage and so is not asked of it. A caching-headers check was added the same day. We considered a compression check and rejected it, because from where our scanner sits we cannot see it reliably, and a check we cannot trust is worse than none.
Link to this entry
A site that turns our scanner away now gets a report, not an error screen
When a firewall or a bot-protection challenge stopped our scanner, we used to end on an error screen. A site that refuses an honest scanner usually refuses the AI crawlers that decide whether you are cited, so that refusal is the finding. The report now says the site is up but turned us away, shows the checks that read the site's published files, and leaves out the checks that need a real page. A banner at the top says so plainly. Unreachable sites and server errors are still errors.
Link to this entry
Broken links, Safe Browsing and a first group of agent readiness checks
The scanner now samples the links on your page for broken and redirecting ones, checks the address against Google Safe Browsing, whether your content is in the HTML rather than built later, whether you have a useful 404 page, how you canonicalise your address, your language and character set, and whether your site serves a clean markdown version to machines. A new group, agent and MCP readiness, covers the emerging ways a site can tell an AI agent how to talk to it, such as OAuth and OpenID Connect discovery. It appears on your report as its own score and stays out of the headline, so a standard that few sites offer yet does not mark every site down.
Link to this entry
The results page grew a viewer, a loader and a way to talk to us
The results page now shows your page's JSON-LD in a panel you can copy from, labels exactly which address was scanned, and links each category card to its breakdown. While a scan runs, a loader shows real progress. A page we could not reach gets its own error screen with a way to retry. You can book a call with us from the end of any report.
Link to this entry
Bot Appétit is live
Paste a page's address and the scanner tells you whether a machine can find, fetch, read, trust and cite it. The first release checks security, SEO, structured data, answer-engine readiness, AI access and links, and turns what it finds into a score, a verdict and a list of what to look at, each finding explained in plain English. Scanning a single page needs no account.
Link to this entry
