CrawlPactMeta-ExternalFetcher
- User-agent token
- Meta-ExternalFetcher
- Purpose
- Agent / action
- Status
- active
- Last verified
- 2026-07-24
Meta documents Meta-ExternalFetcher as a crawler that “fetches individual links at a user’s
request” to support “evaluating and improving agentic AI capabilities — including helping AI
navigate websites to complete tasks for users.” This is a user-directed, single-page fetch rather
than a bulk crawl.
A note on the token
Meta’s own documentation writes the literal user-agent string in lowercase,
meta-externalfetcher/1.1. robots.txt user-agent matching is case-insensitive (RFC 9309
§2.2.1), so this does not change how a Disallow rule applies.
Site-owner controls
Disallowing Meta-ExternalFetcher prevents Meta’s agentic AI features from fetching this
specific page on a user’s behalf — a single-page, user-directed retrieval, not a bulk crawl. It
does not affect Meta-ExternalAgent (bulk AI-training crawling), Meta-WebIndexer (search), or
Meta-ExternalAds (advertising validation), each of which Meta documents as a separate token.
See /limitations for what a robots.txt rule can and cannot guarantee.
Example robots.txt configuration
To disallow Meta-ExternalFetcher specifically, without affecting any other crawler:
User-agent: Meta-ExternalFetcher
Disallow: /If no dedicated User-agent: Meta-ExternalFetcher group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.
Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.
Related crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.
Audit a domain