Preview environment
CrawlPact

AI crawler checker

Enter a domain to see whether documented AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others in CrawlPact's registry — appear allowed or blocked, with the exact matched robots.txt rule and line for each.

What this checks

  • Runs a full CrawlPact scan of the domain and takes you straight to the crawler-by-crawler findings.
  • For each crawler in CrawlPact's registry, shows whether it appears allowed or blocked by the retrieved robots.txt, and the exact matched rule and line.
  • Distinguishes search, training, user-triggered, and agent crawlers from the same operator (for example OpenAI's OAI-SearchBot, GPTBot, and ChatGPT-User), since blocking one does not affect the others.

What this does not check

This reports declared policy and CrawlPact's own retrieved response — not whether any crawler actually obeys it, and not crawlers absent from CrawlPact's registry. See limitations for the full list of boundaries.

Related

Want the complete picture?

This checker is a view into a full CrawlPact audit, which also covers llms.txt, RSL, Content Signals, and your Policy Health Score.

Run the full audit