CrawlPactrobots.txt syntax basics for AI crawler tokens
Published 7/1/2026
robots.txt is a plain-text file at the root of a domain (/robots.txt) made up of one or more
User-agent groups, each containing Allow and Disallow directives, plus optional Sitemap
declarations.
Group matching
A crawler matches the most specific User-agent group that names its token exactly; if no exact
match exists, it falls back to the wildcard (User-agent: *) group. Within a matching group, the
longest matching path rule wins when Allow and Disallow rules conflict for the same path —
this is the RFC 9309 behaviour CrawlPact’s parser follows (see /methodology).
Adding an AI-specific rule
User-agent: GPTBot
Disallow: /This creates a dedicated group for a single crawler token without affecting the wildcard group that other, unlisted crawlers fall back to. CrawlPact’s recommendation engine generates rules in this shape, preserving unrelated existing directives.
Related guides
How to block only AI training crawlers, without blocking AI search
A step-by-step implementation guide for writing robots.txt groups that restrict training-purpose crawlers while leaving search-purpose crawlers untouched.
How to publish an llms.txt file
A step-by-step implementation guide for adding a valid llms.txt (and llms-full.txt) file to your site.
How to publish an RSL (Really Simple Licensing) declaration
A step-by-step implementation guide for declaring permitted and prohibited AI uses of your content via RSL.
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain.
Audit a domain