CrawlPactRobots.txt AI validator
Enter a domain to validate its robots.txt — including duplicate groups, overly broad wildcard blocks, and other syntax issues — with the exact matched rule and line number for each documented AI crawler.
What this checks
- Retrieves
robots.txtand parses it using the same RFC 9309 group-matching logic CrawlPact's scanner uses — longest-path-wins, wildcard-group fallback, and case-insensitive user-agent matching. - Flags duplicate
User-agentgroups for the same token and conflicting rules. - Shows the exact matched rule and line number for each documented AI crawler token, not just whether the file parses.
What this does not check
robots.txt is a voluntary declaration — this cannot confirm any crawler actually honours it, and a syntactically valid file can still be logically wrong for what you intend. See limitations.
Related
- robots.txt syntax basics for AI crawler tokens
- Why a robots.txt rule isn't blocking a crawler
- methodology for the full parsing rules
Want the complete picture?
This checker is a view into a full CrawlPact audit, which also covers llms.txt, RSL, Content Signals, and your Policy Health Score.