Google-Extended: what it is, its user agent, and how to block it

What Google-Extended does

Operator: Google. Job: Model training. Obeys robots.txt.

“Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.”

Google's Google-Extended documentation

Google-Extended user agent

robots.txt token: Google-Extended (matched case-insensitively).

Google-Extended is a robots.txt token, not a crawler: it never appears in a User-Agent header or your logs. Google's main crawler does the fetching, and a Google-Extended group only decides how that data may be used.

robots.txt: allow or block Google-Extended

User-agent: Google-Extended
Allow: /

User-agent: Google-Extended
Disallow: /

Training only, not citations. It controls Gemini training and grounding outside Google Search. AI Overviews and AI Mode use Googlebot, so blocking Google-Extended doesn't remove you from them.

Once a group names Google-Extended, it ignores your User-agent: * rules, so copy any paths you still want blocked into it.

Check your own site

The AI crawler checker reads your robots.txt the way Google-Extended does and tests what your firewall returns. To write the rules, use the AI robots.txt generator.