CrawlPactCCBot
- User-agent token
- CCBot
- Purpose
- Research
- Status
- active
- Last verified
- 2026-07-01
CCBot is operated by the non-profit Common Crawl Foundation to build a large, openly available
web crawl dataset.
Why it matters for AI policy specifically
Common Crawl’s dataset is widely reused as a training source by third parties beyond Common
Crawl itself. A website that blocks every named AI-training crawler but leaves CCBot
unaddressed may still have its content indirectly available for training through downstream
reuse of the Common Crawl corpus — a conflict CrawlPact surfaces explicitly rather than
silently.
Site-owner controls
Disallowing CCBot removes this content from future Common Crawl dataset snapshots going
forward. It has no effect on any downstream party that already holds an earlier snapshot, and no
effect on any other operator’s crawler evaluated separately by CrawlPact (such as GPTBot or
ClaudeBot) — CCBot governs only Common Crawl’s own future crawling, not a broader
AI-training relationship. See /limitations for what a robots.txt rule can and
cannot guarantee.
Example robots.txt configuration
To disallow CCBot specifically, without affecting any other crawler:
User-agent: CCBot
Disallow: /If no dedicated User-agent: CCBot group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.
Verified against the source above as of 2026-07-01 — see how CrawlPact verifies crawler information.
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.
Audit a domain