Everything we check

Answer engines read your HTML and never run your JavaScript. Content that only appears after JavaScript is content ChatGPT, Claude and Perplexity can never quote — even though Google and Microsoft Copilot can see it.

What good looks like The page's main content is already in the HTML the server sends, before any JavaScript runs.

AI engines read the HTML your server sends. A page whose HTML carries almost no readable text gives them almost nothing to quote.

What good looks like The core content is server-rendered and present in the raw HTML.

Real landmarks (header/nav/main/article) tell machines what each region is, so they parse and cite you cleanly.

What good looks like The page uses header, nav, main and article instead of generic divs.

Answer engines parse by passage, not whole page; self-contained sections are far more likely to be lifted.

What good looks like One idea per section (~200–400 words), with descriptive headings, lists and tables.

Declared authorship and entity signals build the E-E-A-T that makes AI confident enough to attribute you.

What good looks like A Person or Organization identified in schema, or an author meta/rel marker.

AI engines favour recently-updated content and use dates to judge relevance — but only a date they can read without guessing.

What good looks like A published and/or last-updated date in schema, article meta, or a time element with a datetime attribute.

llms.txt is a community proposal (llmstxt.org), not a ratified standard: a plain-text Markdown index of your key content at a fixed path, so a machine can see which pages you consider important without inferring it from HTML built for human eyes. We make no claim that any particular assistant reads it; we check for it because it is the one machine-readable map of that kind.

What good looks like A valid /llms.txt: an H1 naming the site, a short summary, then sections of links to the pages that matter.

Serving a clean markdown version to agents that ask for it gives them noise-free content. Nice-to-have; few platforms support it.

What good looks like A clean markdown version of the page is available to agents — either on request, or at the same address with .md on the end.

A program that reads your page, such as a browser agent or a screen reader, finds each control by its name. An icon with no label, or a clickable box that is not a real button, gives it nothing to find.

What good looks like Every button, link and form field has a readable name, and anything clickable is a real button or link.

Blocking the search-and-answer bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) stops those crawlers reading your pages. Fetchers that load a page because a person asked may not check robots.txt at all, so this file decides who may crawl you, not every request.

What good looks like The AI crawlers you want are allowed in robots.txt.

AI answer engines can only cite pages they can load. If your site turns away automated visitors indiscriminately, GPTBot, ClaudeBot and PerplexityBot go with them and you vanish from AI answers. Note the distinction: most bot-protection vendors allow VERIFIED crawlers while refusing unverified ones, so being refused is not by itself proof that AI crawlers are blocked.

What good looks like Your pages load for honest automated clients — AI crawlers aren't blocked at the door.

A page can be indexable and still tell every engine not to quote it. A nosnippet directive keeps your text out of search snippets and out of Google's AI answers, and Bing's noarchive keeps it out of Copilot's answers.

What good looks like No nosnippet, max-snippet:0 or noarchive on pages you want quoted.

An AI assistant that fetches a page while someone waits will stop waiting if it is slow. This is one fetch of your page from our server: a hint of how a live fetch would go, not a promise.

What good looks like The page's HTML arrives within a few seconds.

Naming AI crawlers one by one, rather than leaving them all to a catch-all rule, is a clear, machine-readable statement of which bots may train on your content, which may index it for search, and which may fetch it for a person. It says what you meant; whether those bots can actually read the page is a separate check.

What good looks like An explicit allow/disallow per AI crawler that matches your actual intent.

Machine-readable markup makes your facts eligible for rich results and easier for AI to quote accurately.

What good looks like Schema.org structured data on the page, as JSON-LD, microdata or RDFa.

Broken or incomplete schema is ignored — or worse, distrusted. (We show the snippet + any errors below.)

What good looks like Every JSON-LD block parses cleanly and each entity declares its @context and @type.

A question paired with a direct answer is exactly what an AI answer engine or a search result quotes. FAQPage and QAPage markup label your Q&A content in a machine-readable format that Google and the AI/search crawlers (Bing, Perplexity, RAG crawlers) still parse. Google removed the visual FAQ rich result from search listings in May 2026, but that was only the SERP feature — the markup stays valid, is still read, and the Q&A content itself is what gets cited, so keep your answers genuine and visible on the page.

What good looks like A JSON-LD FAQPage block (multiple questions, your own answers) backing a real, visible FAQ — or, for a page built around one user-answered question, a QAPage block.

sameAs links let engines confirm you're one real entity across sources — the foundation of being named by AI.

What good looks like An Organization entity — JSON-LD or microdata — carrying a logo and sameAs links to your profiles.

Type-appropriate markup (Article, Product, LocalBusiness, Breadcrumb…) unlocks the matching rich results.

What good looks like Schema types that match the page's purpose, with required fields filled.

Some AI fetchers read the page text and never see JSON-LD, so a price, date or address that lives only in your markup may never reach them.

What good looks like The key facts in your structured data — name, price, date, author, address, phone — also appear in the visible text.

A BreadcrumbList states where a page sits in your site's hierarchy, in a form a machine can read directly instead of inferring it from the URL or the navigation.

What good looks like A valid BreadcrumbList (JSON-LD or microdata) on interior pages, matching the on-page breadcrumb trail.

Each redirect hop is another request an AI client or crawler has to make before it reaches your content, and some give up. A single clean redirect is normal; a chain of them is latency you are adding to every visit. A page that does not answer at all is reported separately, as a refusal rather than a score.

What good looks like The URL answers directly, or through one clean redirect.

A Cache-Control, ETag, or Last-Modified header lets browsers and CDNs skip re-downloading an unchanged page — faster repeat visits, less server load, and a small hand to crawlers hitting you often.

What good looks like The page sends a Cache-Control, ETag, Last-Modified, or Expires header.

Serving the same content on both www and non-www (or http and https) splits ranking authority.

What good looks like All variants 301 to one preferred host.

A stray Disallow: / can hide your whole site from search; a missing robots.txt leaves crawling ambiguous.

What good looks like A valid robots.txt that allows your public pages and links to the sitemap.

A meta robots or X-Robots-Tag noindex removes the page from search entirely — often left on by mistake after launch.

What good looks like No noindex on pages you want found.

The canonical consolidates ranking signals onto the URL you want and resolves duplicate-content ambiguity. It counts whether you declare it as a link element or an HTTP Link header — Google treats both the same.

What good looks like A self-referencing canonical pointing to the preferred, on-domain URL — as a <link rel="canonical"> element or a Link: rel="canonical" response header.

The title is the strongest on-page relevance signal and the first thing a search engine or an answer engine reads to know what a page is. A page with no title is unnamed to a machine.

What good looks like Every page has a unique, keyword-led title element.

A meta description is a short, author-written summary in a fixed place that search engines, answer engines and link previews all read. A page without one leaves a machine to summarise it by guesswork.

What good looks like A unique description that sums up the page, declared with the name attribute.

The H1 states the page topic to crawlers and AI. Without one, a machine has to guess what the page is about.

What good looks like An H1 that states the page's topic.

A clean H1→H2→H3 structure is how crawlers, screen readers, and AI parse your content.

What good looks like Headings nest in order with no skipped levels.

When the title, H1, URL, and opening line agree on the topic, relevance is unambiguous.

What good looks like The main topic appears in the title, the H1, the slug, and the first paragraph.

Alt text powers image search, gives screen-reader context, and helps machines understand the page.

What good looks like Meaningful images carry concise, descriptive alt text.

Google's crawler reads only the first 2 MB of a page's HTML and ignores the rest, so anything after that point cannot be indexed from this page. A smaller document is also less to download for any machine.

What good looks like The HTML document is comfortably inside what a crawler will read.

OG tags control the preview card when your links are shared or cited — a growing discovery surface.

What good looks like og:title, og:description, og:image and og:url are all set, declared with property=.

A sitemap speeds discovery and indexing, especially for newer pages with few inbound links.

What good looks like A valid /sitemap.xml, referenced from robots.txt, whose URLs resolve.

A soft-404 (a 200 for a missing page) confuses crawlers and can get junk indexed.

What good looks like Unknown URLs return a true 404 with a helpful page.

A lang attribute tells a machine which language the text is in, and a declared character set tells it how to decode the bytes. Without them a parser has to guess, and a guess can garble non-English text.

What good looks like A lang attribute on the html element and a declared character set, in a meta charset tag or the Content-Type header.

hreflang serves the right language/region version and prevents duplicate-content confusion.

What good looks like Valid, reciprocal hreflang tags across your language variants.

A <meta http-equiv="refresh"> redirect only happens after the page has loaded, so a machine that reads the response sees the page you are leaving, not the one you are sending it to. A server redirect tells every client where to go before it reads any content.

What good looks like Redirects are server redirects (301 or 302), not a meta refresh.

Dead links waste crawl budget, leak authority, and frustrate readers.

What good looks like On-page links resolve (no 4xx/5xx).

Linking through redirects (or chains) wastes crawl budget and slightly dilutes authority.

What good looks like Internal links point straight at final URLs.

Descriptive anchors tell engines (and readers) what's on the other end; 'click here' tells them nothing.

What good looks like Meaningful anchor text instead of 'click here' / bare URLs.

A crawler follows a link only if it finds an address in it. A link built from a script, or an anchor with no address, leads nowhere for a machine, and the pages behind it can go unfound.

What good looks like Every link is an anchor with a real web address.

HTTPS is a baseline trust and ranking signal; browsers warn users away from insecure pages, and AI engines skip them.

What good looks like Every page loads over HTTPS with a valid, unexpired certificate.

If the insecure http:// version still loads, you split trust and risk duplicate content.

What good looks like http:// requests 301-redirect to the https:// version.

Browsers outright block insecure scripts, styles and frames on an https page — breaking the page — and auto-upgrade insecure images/media, which then break if the host doesn't serve https.

What good looks like Every script, style, frame, image and media file is referenced over https.

A site flagged by Google Safe Browsing is hidden behind warnings and won't be recommended.

What good looks like The site is clean in Google Safe Browsing.

Content-Signal directives (search / ai-input / ai-train) let you state how your content may be used by AI.

What good looks like Clear Content-Signal directives that reflect your policy.

A Link header (RFC 8288) can point an agent at your sitemap, a description of your site or your API documentation from the response alone, without parsing any HTML. It is optional and uncommon. No standard names llms.txt in a Link header, so we do not look for that.

What good looks like A Link header whose relation names a discovery resource you publish, such as sitemap, service-doc, service-desc, api-catalog or describedby.

Advertises an MCP server so agents can discover and use your tools before they connect. Only relevant if you expose agent-facing infrastructure.

What good looks like An AI catalog (/.well-known/ai-catalog.json) with an entry for your MCP Server Card, as the final MCP specification describes. Cards published at the older well-known paths still count, and the result says they are the legacy route.

A single index of the AI-facing things you publish — MCP servers, agent cards, skills, datasets — so an agent finds them in one hop instead of guessing at paths. Adoption is early and the format is still settling, so not having one is normal today rather than a defect.

What good looks like An AI catalog that actually lists something, found at /.well-known/ai-catalog.json or /.well-known/ard.json, or pointed to from robots.txt (an Agentmap line), a link element in the page head, or a Link header. An index with nothing in it tells an agent no more than having none.

An A2A Agent Card is the business card another AI agent reads before it talks to yours: who you are, what skills you offer and how to connect. Only relevant if you run an agent that other agents are meant to call.

What good looks like A JSON Agent Card at /.well-known/agent-card.json that names the agent and lists its skills. The older /.well-known/agent.json is read too.

WebMCP lets a page offer its real actions, like searching or booking, to an AI agent running in the browser, instead of the agent guessing at buttons. Forms can declare themselves as tools right in the HTML. It is an experimental browser feature, so most sites have nothing here yet.

What good looks like Forms that declare a tool name and description in the markup, or the WebMCP bridge script, so a browsing agent can use the page's actions directly.

The IETF's AI preferences work gives a site a standard way to say, in robots.txt or a response header, whether its content may be used for AI training, AI answers or search. Nobody is known to honour it yet, so this is a statement of intent rather than a lock.

What good looks like A Content-Usage line in robots.txt, or a Content-Usage response header, stating your preference for training, AI use and search.

Really Simple Licensing lets a publisher point machines at machine-readable licence terms for its content, including terms that ask AI companies to pay. It is backed by several large CDNs and publishers, though no AI company is known to honour it yet.

What good looks like A License line in robots.txt, or a licence link of the RSL type in the page head or a Link header, pointing at your RSL terms.

TDMRep is a W3C community standard for reserving the right to text-and-data mining of your content, used by publishers in the EU where that right can be opted out of. It can be a response header, a meta tag or a small file.

What good looks like A tdm-reservation signal of 0 or 1 as a response header, a meta tag, or a /.well-known/tdmrep.json file.

Some AI agents never ask a site for Markdown; they follow a link to it. A link in the page head to the Markdown version is how those agents find the cleaner copy of the page.

What good looks like A link in the page head, with the relation alternate and the type text/markdown, pointing at the Markdown version of the page.

If robots.txt says one thing about AI training and the page or its Markdown copy says another, an AI client may follow the more permissive one. Some CDNs add a permissive default to Markdown responses when the site sent none, so the contradiction can appear without anyone writing it.

What good looks like The same content signals in robots.txt, in the page's response headers and on its Markdown version.

A discoverable manifest of skills an agent can use on your site.

What good looks like A valid Agent Skills index at /.well-known/agent-skills/index.json.

A machine-readable index of your APIs (RFC 9727) lets agents find and use them.

What good looks like A valid /.well-known/api-catalog.

Discovery metadata lets agents learn how to authenticate against protected resources.

What good looks like Valid OAuth authorization-server metadata at the well-known path.

RFC 9728 protected-resource metadata is how an MCP server tells an agent which authorization server guards it — the entry point of the MCP auth handshake.

What good looks like A valid /.well-known/oauth-protected-resource pointing at your authorization server.

OIDC discovery metadata lets agents and clients learn your token, authorization and JWKS endpoints without hard-coding them.

What good looks like Valid OpenID Connect discovery metadata at /.well-known/openid-configuration.

Web Bot Auth (IETF draft) lets a BOT OPERATOR cryptographically prove its traffic: the operator publishes signing keys on its own domain, and websites verify them. This one only applies if this domain operates a bot or agent — a normal publisher site has nothing to publish here, and that's expected.

What good looks like Only if you operate a signed bot or agent: a JWKS at /.well-known/http-message-signatures-directory on the domain your bot signs as, so the sites it visits can verify it.

UCP (Google) is how an AI shopping agent finds out what you sell without a human browsing your site. The agent appends /.well-known/ucp to your domain and reads a JSON manifest of your services, capabilities, payment handlers and API endpoints. Only relevant if you sell something — we don't check it otherwise.

What good looks like A JSON manifest at /.well-known/ucp declaring what you sell, how to transact, and which transports you support.

OAuth tells an agent how to authenticate. auth.md tells it how to register in the first place — the step before that. Without one, an AI agent acting for a customer has no documented way to get credentials for your service.

What good looks like A /auth.md describing your registration flows, scopes and endpoints, alongside your OAuth protected-resource metadata.

Two competing drafts let a domain announce its AI agent endpoints in DNS: AID, a community specification, and DNS-AID, an IETF draft. Neither is a ratified standard and they are not compatible with each other, so a missing record is expected.

What good looks like If you run an agent-facing service: a record in either draft's format that points to an endpoint that answers.

Headers like HSTS, X-Content-Type-Options and a referrer policy harden the site and are a recognised best-practice signal.

What good looks like HSTS, X-Content-Type-Options, X-Frame-Options/CSP frame-ancestors, and a Referrer-Policy are present.

Headers that reveal exact server/framework versions hand attackers a head start.

What good looks like Server and X-Powered-By headers are absent or generic.

A /.well-known/security.txt (RFC 9116) gives security researchers a standard way to report a vulnerability — a maturity signal shipped by Google, GitHub and Meta.

What good looks like A /.well-known/security.txt with the two fields RFC 9116 requires: Contact: and a future Expires: date.

Without a viewport tag a phone renders the page at a pretend desktop width and shrinks the result to fit, so none of your responsive breakpoints ever fire and the page arrives as an unusable miniature.

What good looks like A width=device-width viewport meta tag.

A title outside roughly 30-60 characters either gets truncated in a search result or is too thin to say what the page is. It's a presentation detail rather than a machine-readability one, which is why it's weighted lightly — the title existing is what actually matters.

What good looks like A title of roughly 30-60 characters — long enough to be specific, short enough to show in full.

A description outside roughly 70-160 characters is either cut off in a search result or too thin to say what the page is. It is a presentation detail rather than a machine-readability one, which is why it does not score. The description existing is what matters.

What good looks like A description long enough to be specific and short enough to show in full.

Without a card + image, links render as a small, less clickable preview on X.

What good looks like twitter:card, title, description and image are set.

Heavy, unsized images are the most common cause of slow loads and layout shift.

What good looks like Modern formats (WebP/AVIF), lazy-loading below the fold, and explicit width/height.

You can't improve what you don't measure; no analytics means no baseline and no lead attribution.

What good looks like GA4 (or a privacy-first equivalent) is installed, ideally via GTM.

Verifying the site is your direct line to index status, errors and search performance.

What good looks like Google Search Console (and ideally Bing) verification is in place.

Your favicon shows next to your result in search and in some AI answer interfaces — a small but real trust and brand signal.

What good looks like A <link rel="icon"> (or apple-touch-icon) pointing at a real icon.

A claim an answer engine can check is one it can repeat and attribute with confidence. Figures stated as figures, a source quoted as a quotation and links out to where a claim comes from are the parts of an article it can lift cleanly. In the one controlled study of generative engines, adding each raised how often a source was used. It is an indicator, not a guarantee.

What good looks like Statistics written out, a quoted source set as a quotation, and links to the original sources of your claims.

Retrieval systems match a reader's question to a section of a page through its heading, so a heading that asks what the section answers sits closest to the query. The evidence is mixed: some large studies find question-style headings cited more, others less, and Google says no special writing is needed. That is why this is shown and not scored.

What good looks like Section headings that read as the question the section answers, with the answer straight after.

An AI reads a page in tokens, and every tool that fetches one has a ceiling on how many it will take in before it cuts the page short. A page that costs a great deal to read risks losing its end, or being skipped for a cheaper source. A site that serves a clean Markdown version to agents that ask lowers that cost, and we judge the cheaper of the two.

What good looks like A page whose text fits comfortably within what a fetching tool will read, or a Markdown version for agents that ask for one.

Text hidden from visitors but left in the page for a machine to read is a known way of trying to steer an AI model that summarises the page. A model cannot tell that such text was not meant for it, so the safest page is one that says nothing to a machine that it does not also say to a visitor. We only report text that is both hidden and worded as an instruction to a model.

What good looks like Everything a machine can read on the page is something a visitor can see, or is ordinary markup such as menus, accordions and data for search engines.

The date beside each address in a sitemap is how a crawler decides what to fetch again. It only helps when it is real: the date the page last changed, in the standard format, never in the future, and not one date stamped on every page. Google says it uses the date only when it proves consistently accurate.

What good looks like Each sitemap entry carries the date its page last changed, in the standard date format, and the dates differ as the pages do.

An llms.txt file exists to send an AI to the pages that matter, so a link in it that leads nowhere is the file failing at its one job. We try a small sample of the links. A slow answer, a rate limit or a bot block counts as not measured, never as broken.

What good looks like Every link in llms.txt opens the page it names.

A reasonable internal-linking structure spreads authority; excessive or all-nofollow outbound links are a smell.

What good looks like Healthy internal linking and a sensible nofollow ratio.