AI crawlers › Google-Extended
Google-Extended: a control, not a crawler
Google-Extended is the odd one on this list: it is a name you put in robots.txt, not a robot that visits you. You will never see it in a log file, because it does not make requests.
Usage control, not a crawler · operator: Google · every fact below is quoted from Google, “List of Google’s common crawlers” and links to it. Nothing on this page is inferred.
what the operator documents
The facts, and where each one comes from
| Fact | What the operator says | Source |
|---|---|---|
| Operator | Google. | Google docs |
| Name to match in robots.txt | Google-Extended | Google docs |
| It is a control, not a crawler | Google documents it as a robots.txt user agent used to manage how content is used. It has no separate request string of its own: the fetching is done by Google’s existing crawlers, so there is nothing new to look for in your logs. | Google docs |
| What it controls | Whether content crawled from your site may be used for training future Gemini model generations, and for grounding — supplying content at prompt time — in Gemini apps and in grounding with Google Search on Vertex AI. | Google docs |
| Effect on Search, in Google's words | “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” | Google docs |
| Documented example | Google’s own example allows one path inside a disallowed directory, which is the pattern to copy if you want to expose exactly one section. | Google docs |
What the operator does not publish
- Any effect on Googlebot. Blocking Google-Extended does not remove you from Search and does not stop Googlebot crawling; the documentation says so directly, and this is the single most common misunderstanding we are asked about.
allow or block
The exact robots.txt lines
Paste these into the robots.txt file at the root of your site. Groups are matched by name, so a group for one robot has no effect on another.
Opt out of the uses it controls
User-agent: Google-Extended
Disallow: /Allow it explicitly
User-agent: Google-Extended
Allow: /The pattern from Google’s own example: one allowed path inside a blocked directory
User-agent: Google-Extended
Allow: /archive/1Q84
Disallow: /archive/Because this is a control rather than a crawler, there is no crawl rate to manage and nothing to verify by address. The only question is whether the group is present and correctly spelled.
what it means for you
What allowing or blocking it actually changes
Blocking it costs you
Grounding eligibility in the products Google names — the case where an assistant pulls your content in at prompt time. If you want to be quotable in those surfaces, blocking it works against you.
Blocking it does not cost you Search
Google states plainly that it is neither an inclusion nor a ranking factor in Search. If somebody told you otherwise, that is the sentence to read them.
The measurement that matters either way
Grounding only helps if the fetched page answers the question. In our twelve-site baseline, agents could not determine terms, contract or warranty in 83% of visits (Note 27). Permission is not an answer.
And the lever that is not in this file at all
Permission is one job; being legible when the robot arrives is another. Google’s page-quality tooling now audits websites for agent-readiness, and the package passes Google’s agent-readiness audit — which is a claim about the surfaces on your site, not about who you let in. We make no claim that any of it improves your search position.
The distinction that decides everything on this page
| Crawlers like this one | A model with a fetch tool | |
|---|---|---|
| Reads robots.txt | Always, and first | Never, in our runs |
| What it is doing | Indexing, then leaving | Answering somebody’s question now |
| What reaches it | Your directives | The bytes of the page it fetched |
Our own server logs plus six models per site across two client sites: the crawlers requested robots.txt before anything else, every time, and not one fetch-tool model requested it once. Source: MARKETING_NOTES_AGENT_VISIBILITY.md, Note 04.
the rest of them
The other crawlers people ask us about
- ClaudeBot — Anthropic, training crawler
- GPTBot — OpenAI, training crawler
- OAI-SearchBot — OpenAI, search crawler
- PerplexityBot — Perplexity, search crawler
Definitions for the vocabulary on this page: AI crawler, answer engine, llms.txt, fact block.
The separate question of whether your site is legible when one of these arrives is graded by Google’s own tooling now: our explainer on the Agentic Browsing audit covers every check it runs.
free audit
See what AI sees on your site
Blocking and allowing is housekeeping. The question underneath it is whether the page a robot reaches actually answers what your buyers ask.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.