CrawlPactClaude-SearchBot
- User-agent token
- Claude-SearchBot
- Purpose
- Search
- Status
- active
- Last verified
- 2026-07-24
Anthropic documents Claude-SearchBot as a crawler that “navigates the web to improve search
result quality for users,” analyzing content specifically to enhance the relevance and accuracy
of search responses — distinct from ClaudeBot’s training purpose.
Why the distinction matters
A website that wants to remain discoverable through Claude’s search features while restricting
training use of its content needs to treat Claude-SearchBot and ClaudeBot as separate
decisions in robots.txt.
Site-owner controls
Anthropic’s own documentation gives this example for blocking any of its crawlers:
User-agent: Claude-SearchBot
Disallow: /See /limitations for what a robots.txt rule can and cannot guarantee.
Example robots.txt configuration
To disallow Claude-SearchBot specifically, without affecting any other crawler:
User-agent: Claude-SearchBot
Disallow: /If no dedicated User-agent: Claude-SearchBot group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.
Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.
Related crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.
Audit a domain