AI Agent Readiness Checker
Test how discoverable and accessible your website is to AI crawlers and agent protocols.
Based on: RFC 9309 — Robots Exclusion Protocol · Google Search Central — Crawlers +7 more
About this validation
Checks how discoverable and accessible a website is to AI crawlers and autonomous agents: robots.txt rules for AI-specific User-Agents, sitemap discovery, and a growing set of emerging, mostly optional discovery protocols (agent cards, MCP, Web Bot Auth, and others). Individual checks cite their own source, since this tool combines several independently-governed mechanisms.
- Protocol / standard:
- Search engine behavior:
- Vendor documentation:
- Reference documentation:
- Rules reviewed:
- Sep 8, 2026
SEOTechTests methodology: Data source: a live fetch of the domain's robots.txt, sitemap, and a set of well-known discovery URLs performed by SEOTechTests. This is a technical diagnostic of what's detectable, not a certification -- many of the checks cover standards that are still evolving and entirely optional for a typical website.
Standards and vendor-documented behavior can change. This reflects the source as last reviewed above, not a permanent guarantee.
FAQ
What is AI agent readiness?
How easily an AI crawler or autonomous agent can discover, access, and understand a website using publicly documented conventions -- robots.txt rules, sitemaps, and a growing set of emerging discovery protocols. It's a technical diagnostic, not a certification, and several of the checks here cover standards that are still evolving and entirely optional.
How do AI crawlers discover a website?
Most AI crawlers follow the same basic discovery path as search engine crawlers: they check robots.txt for rules addressed to their specific User-Agent, look for a sitemap to find URLs, and request pages directly. Some also probe well-known URLs for machine-readable metadata -- agent cards, API catalogs, protocol descriptors -- that describe how to interact with a site programmatically.[1]
How are AI crawlers different from search engine crawlers?
Search crawlers like Googlebot primarily build a search index. AI crawlers serve different purposes -- some fetch content live to answer a user's query (e.g. ChatGPT-User), others collect content for model training (e.g. GPTBot, Google-Extended). robots.txt access, model-training permission, and search indexing are three separate concerns controlled in different ways: blocking a crawler in robots.txt does not affect training data already collected, and allowing a crawler does not affect search indexing.[2, 3]
How does robots.txt control AI crawler access?
Several AI companies publish and respect distinct User-Agent tokens for their crawlers -- GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, and others. A site can allow, disallow, or leave unaddressed each one independently. Leaving an agent unaddressed is not the same as blocking it: most crawlers fall back to whatever rule applies to User-agent: *, or crawl freely if no rule applies at all.[1, 3, 4, 5]
What is agent protocol discovery (agent cards, MCP, A2A)?
A newer category of well-known URLs lets a site advertise machine-readable capabilities: an API catalog, OAuth metadata for authenticated access, an MCP server descriptor, or an Agent2Agent (A2A) Agent Card. These are mostly relevant to sites that expose an API or an agent-facing service -- a typical content website has no need for most of them, and not having one is not a defect.[6, 7]
What are these emerging AI web standards (Markdown negotiation, Web Bot Auth, agentic commerce)?
Markdown content negotiation, TDMRep content signals, Web Bot Auth, WebMCP, Agentic Resource Discovery, and agentic commerce protocols (x402, MPP, UCP, ACP) are all genuinely new and, in most cases, unratified or vendor-associated conventions. This checker reports whether it detected each one, but their absence does not indicate a problem, and implementing them is not a requirement for search ranking or general web compatibility.[8, 9]