CrawlPactMeta-WebIndexer
- User-agent token
- Meta-WebIndexer
- Purpose
- Search
- Status
- active
- Last verified
- 2026-07-24
Meta documents Meta-WebIndexer as a crawler that “navigates the web to improve Meta AI search
result quality for users” by analysing content to enhance relevance and accuracy — a
search-purpose crawler, distinct from Meta-ExternalAgent’s training purpose.
A note on the token
Meta’s own documentation writes the literal user-agent string in lowercase,
meta-webindexer/1.1. robots.txt user-agent matching is case-insensitive (RFC 9309 §2.2.1), so
this does not change how a Disallow rule applies — CrawlPact displays crawler names in the
capitalisation convention used consistently across this directory.
Site-owner controls
Disallowing Meta-WebIndexer affects this content’s relevance and accuracy in Meta AI’s search
results specifically. It does not affect Meta-ExternalAgent (AI training), Meta-ExternalAds
(advertising validation), or Meta-ExternalFetcher (user-triggered agent fetches) — each is
governed by its own separate token. See /limitations for what a robots.txt rule
can and cannot guarantee.
Example robots.txt configuration
To disallow Meta-WebIndexer specifically, without affecting any other crawler:
User-agent: Meta-WebIndexer
Disallow: /If no dedicated User-agent: Meta-WebIndexer group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.
Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.
Related crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.
Audit a domain