CrawlPactGoogle-CloudVertexBot
- User-agent token
- Google-CloudVertexBot
- Purpose
- Agent / action
- Status
- active
- Last verified
- 2026-07-24
Google’s own crawler documentation describes Google-CloudVertexBot as affecting “crawls
requested by site owners for building Vertex AI Agents.” Unlike Googlebot or Google-Extended,
this crawler acts on a specific site owner’s own request, rather than a general web-wide sweep.
Why this is categorised as “agent”
Google’s documentation frames this crawler around a site owner’s own request to build an AI agent against their content, which fits CrawlPact’s registry’s “agent” purpose category more than a general search or bulk-training crawl.
Site-owner controls
Disallowing Google-CloudVertexBot prevents crawls performed specifically because a site owner
(not necessarily you) requested that a Vertex AI Agent be built against this content. It does not
affect Google Search indexing (Googlebot) or the generative-AI-training opt-out
(Google-Extended) — this token governs only agent-initiated, request-driven access, not a
general web-wide crawl. See /limitations for what a robots.txt rule can and
cannot guarantee.
Example robots.txt configuration
To disallow Google-CloudVertexBot specifically, without affecting any other crawler:
User-agent: Google-CloudVertexBot
Disallow: /If no dedicated User-agent: Google-CloudVertexBot group exists in a domain's robots.txt, this crawler falls back to whatever the wildcard User-agent: * group says (RFC 9309) — see robots.txt syntax basics for how group selection works.
Verified against the source above as of 2026-07-24 — see how CrawlPact verifies crawler information.
Related crawlers
See how this applies to your own site
Run a free audit to check your declared AI crawler policy against your own domain, or use the AI crawler checker to check this one crawler specifically.
Audit a domain