Robots.txt Checker
Enter a domain to fetch its robots.txt and review the parsed rules.
Based on: RFC 9309 — Robots Exclusion Protocol · Google Search Central — robots.txt +3 more
About this validation
Validates robots.txt syntax, crawler groups, and Allow/Disallow rules against the Robots Exclusion Protocol (RFC 9309), and flags directives -- Crawl-delay, Host, Clean-param, and the unofficial Noindex -- whose support currently differs between Google, Bing, and Yandex. Also includes a URL tester that applies the same longest-match precedence a compliant crawler would, to show whether a given path is actually allowed or blocked.
- Protocol / standard:
- Search engine behavior:
- Rules reviewed:
- Sep 8, 2026
SEOTechTests methodology: Data source: a live fetch of the domain's /robots.txt performed by SEOTechTests, parsed against RFC 9309's group and precedence rules. Engine-specific nuances (which directives each engine honors) are verified against that engine's own current documentation rather than generic SEO guidance, since third-party summaries are frequently out of date on this point.
Standards and vendor-documented behavior can change. This reflects the source as last reviewed above, not a permanent guarantee.
FAQ
Do Google, Bing, and Yandex all interpret robots.txt the same way?
They share the same baseline -- User-agent, Allow, Disallow, and Sitemap, as defined by RFC 9309 -- but several extra directives are only honored by some of them. This checker highlights those cases directly against each group's User-agent line rather than assuming one engine's behavior applies everywhere.[1, 2, 4]
Does Google honor the Crawl-delay directive?
No. Google has never supported Crawl-delay in robots.txt; setting it has no effect on Googlebot. Crawl rate for Google is controlled separately, through Search Console's crawl rate settings.[2]
Does Yandex still honor Crawl-delay?
No, not since February 22, 2018. Yandex's own documentation states it no longer takes the Crawl-delay directive into account; a site should use the crawl-rate setting in Yandex Webmaster instead. Bing is the one major engine that still documents honoring Crawl-delay, within a 1-30 second range.[5, 3]
What are the Host and Clean-param directives?
Both are Yandex-specific extensions, not part of RFC 9309. Clean-param tells Yandex which URL parameters to ignore when indexing and is still current. Host used to indicate a site's preferred mirror, but it no longer appears in Yandex's own current directives documentation -- treat it as deprecated. Neither has any effect on Google, Bing, or other crawlers.[4]
Can a Noindex line in robots.txt keep a page out of search results?
No. Noindex in robots.txt was never an official part of the standard -- it was an undocumented behavior Google unofficially honored, and Google stopped supporting it in September 2019. To keep a page out of the index, use a per-page noindex meta tag or an X-Robots-Tag HTTP header instead.[2]
How does the URL tester decide whether a path is blocked?
It looks for the group that explicitly names the chosen crawler, falling back to the wildcard "*" group if there isn't one, then applies RFC 9309's longest-match-wins rule across that group's Allow/Disallow patterns -- with a tie going to the less restrictive (Allow) rule, matching how compliant crawlers resolve conflicting rules of equal length.[1]