Anthropic

ClaudeBot

User-agent token: ClaudeBot
Purpose: Training
Status: active
Last verified: 2026-07-24

ClaudeBot is documented by Anthropic as the crawler used to gather publicly available web content for training its Claude models.

Site-owner controls

Anthropic documents standard robots.txt support for disallowing ClaudeBot. As with other training-purpose crawlers, blocking it is a declared-policy signal only — see /limitations for what a robots.txt rule can and cannot guarantee.

Example robots.txt configuration

To disallow ClaudeBot specifically, without affecting any other crawler:

User-agent: ClaudeBot
Disallow: /

If no dedicated User-agent: ClaudeBot group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.

Official source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.

Related guides

ClaudeBot vs. Claude-User vs. Claude-SearchBot: which should you block?
Anthropic operates three separate crawler tokens for training, user-triggered retrieval, and search. A decision guide for configuring each independently.

Related crawlers

See how this applies to your own site

Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.

Audit a domain