CrawlPactMethodology
What CrawlPact evaluates
CrawlPact retrieves a bounded set of public resources — robots.txt,llms.txt, RSL declarations, Content Signals, relevant HTML meta and HTTP headers, and sitemap declarations — and evaluates them against a versioned registry of documented AI crawlers.
robots.txt evaluation
The parser follows RFC 9309 behaviour as closely as practical: a crawler matches the most specific User-agent group naming its token, falling back to the wildcard group; within a group, the longest matching path rule wins between conflicting Allow/Disallow directives. Original text, line numbers, and group boundaries are preserved so every result can cite its exact evidence.
Declared policy vs. observed response vs. actual behaviour
Every report distinguishes three things: the declared policy (what a website asks crawlers to do), the observed response (what CrawlPact successfully retrieved), and actual crawler behaviour (what an external crawler truly does — which a public configuration audit cannot prove). See limitations.
Determinism
Findings and recommendations are generated by rule, not by an AI model. The same input, registry version, and ruleset version always produce the same result — this is what makes change monitoring meaningful: a different result implies either the target changed (website drift) or CrawlPact's registry changed (registry drift), never randomness.
Registry verification
A crawler is not published to the registry without a reliable official source, a verified user-agent token, a purpose classification, a verification date, and administrator approval. Registry releases are immutable once published; historical scans retain the registry version they were evaluated against.
What each signal can and can't tell you
Every signal CrawlPact retrieves is a declaration, not a guarantee. The table below states plainly what CrawlPact can and cannot conclude from each one, and how mature the underlying specification actually is — a signal isn't more trustworthy just because CrawlPact checks it.
| Signal | CrawlPact can infer | CrawlPact cannot infer | Specification maturity |
|---|---|---|---|
robots.txt | Whether a specific crawler token appears allowed or disallowed for a given path, per the retrieved file's own rules. | Whether the crawler actually honours it, or whether the same file is served identically to every requester (some sites vary responses by IP or user-agent). | RFC 9309 — IETF Proposed Standard (Sept 2022) |
Meta robots / X-Robots-Tag | The indexing/following instruction declared for the specific page(s) checked. | Whether the same directive applies sitewide — checks are bounded, not a full-site crawl. | Widely supported convention documented by major search engines, not an IETF/W3C standard. |
llms.txt | Presence, file structure, and linked resources. | Whether any AI system reads or acts on it — a missing file carries no implication either way. | Independent community proposal (first published Sept 2024) — not an IETF or W3C standard. |
| RSL | Declared permitted/prohibited uses and any stated payment terms. | Enforcement — RSL is a machine-readable declaration, not a technical control, and no crawler is required to honour it. | RSL 1.0 (published Dec 2025), governed by an industry technical steering committee — not an IETF/W3C standard. |
| Content Signals | Declared search/ai-train/ai-input values. | Real-world effect — as of this writing, no major crawler operator is known to act on this signal. | A single vendor's proposal (Sept 2025), not ratified. A separate IETF effort (AI Preferences) is still in draft. |
| Sitemap | Which URLs the site declares, from a bounded sample. | Full site coverage — CrawlPact samples a sitemap, it does not crawl an entire site. | Long-standing multi-engine convention (sitemaps.org), not an IETF/W3C standard. |
Content and registry corrections
If you find an incorrect crawler token, operator, or purpose classification; an outdated official source; a broken official link; incorrect guide or platform-instruction content; a misleading report explanation; missing uncertainty; or a typographical or accessibility issue, report it to support@crawlpact.com. Include the affected URL, a description, a supporting official source where possible, and your suggested correction.
Each submission is reviewed:
- The submission is received and logged.
- Supporting evidence is reviewed, with official sources prioritised over unverified claims.
- Classification uncertainty is recorded explicitly where it exists.
- The registry or content is updated through normal editorial review, where the correction is confirmed.
- Material registry changes are reflected in the changelog where supported.
Not every submission results in a change, and submissions are not accepted as automatic registry truth — each is weighed against official operator documentation. CrawlPact does not promise a fixed response time for corrections.
Last substantive update: 31 July 2026.