AI Bot Access Checker
See which AI crawlers robots.txt allows or blocks, and which rule decides it.
Resolves each major AI crawler against the site's robots.txt and shows whether it is allowed by a rule of its own, by the wildcard group, or by no rule at all. This covers crawler access only: it says nothing about model-training permission or search indexing, which are governed separately.
This is one of the checks the AI Agent Readiness Checker scores under Bot Access Control, and it can also be run on its own for any URL.
Based on: RFC 9309 — Robots Exclusion Protocol · Google Search Central — Crawlers +3 more
About this validation
Resolves AI crawler User-Agents against robots.txt using RFC 9309 matching rules, including the wildcard fallback. Crawler access is all robots.txt governs -- it carries no model-training or indexing permission.
- Protocol / standard:
- Search engine behavior:
- Vendor documentation:
- Rules reviewed:
- Nov 11, 2026
Standards and vendor-documented behavior can change. This reflects the source as last reviewed above, not a permanent guarantee.