Google doesn’t use a separate crawler to collect content for its AI. Googlebot crawls for Google Search, and Search’s AI features, such as AI Overviews, are part of Search. Google-Extended is a robots.txt token, not a crawler: it lets you say whether your content may be used to train Gemini models and to ground Gemini’s answers, without affecting how you appear in Google Search.
Googlebot: Search, including AI Overviews
Googlebot fetches pages for Google Search. Search’s AI features use the Search index, so there isn’t a separate “AI Overviews bot” to block. If you block Googlebot, you leave Google Search altogether, AI features included. For almost every website, that’s not what you want.
To keep a page out of AI Overviews specifically, Google’s tools are the ones for Search snippets, such as nosnippet, which also change how the page looks in normal results. Check Google’s Search documentation before using them.
Google-Extended: Gemini training and grounding
Google-Extended has no user-agent of its own; Google crawls with its existing crawlers and reads this token to decide how the content may be used. Disallowing it tells Google not to use your content to train future Gemini models or to ground answers in Gemini apps and Vertex AI.
User-agent: Google-Extended
Disallow: /What this doesn’t do:
- It doesn’t remove you from Google Search.
- It doesn’t change your ranking in Search.
- It doesn’t affect AI Overviews, which are part of Search.
Because there is no Google-Extended crawler, you will never see “Google-Extended” in your logs.
The other Google fetchers you may see
| Name in the user-agent | What it is |
|---|---|
| Googlebot | Google Search (desktop and smartphone) |
| GoogleOther | A generic crawler Google uses for internal research and development |
| Google-InspectionTool | Search Console’s testing tools, when you inspect a URL |
| Google-CloudVertexBot | Crawls a site when its owner sets up a Vertex AI agent for it |
| Google-NotebookLM | Fetches a page when a NotebookLM user adds it as a source |
Fetchers like Google-NotebookLM work because a person asked, not on their own schedule. Google publishes their addresses in a separate list of user-triggered fetchers.
How to verify a Google visit
Google offers two ways, and either is enough:
- IP lists. Google publishes JSON lists for Googlebot, its special-case crawlers, and user-triggered fetchers. The address must be in the right one.
- Reverse DNS, confirmed both ways. The address must resolve to a name ending in
googlebot.com,google.comorgoogleusercontent.com(depending on the crawler), and that name must resolve back to the same address.
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1“Googlebot” is one of the most faked names on the web, because almost every site lets it in. Paste any visit into the bot checker or see the Googlebot page. The general method is in how to verify any AI crawler.
People who come from Gemini
Visitors who click a link in a Gemini answer usually arrive from gemini.google.com. Clicks from AI Overviews, on the other hand, come from Google Search itself, so analytics counts them as ordinary organic search. See how to find AI traffic in your analytics.
What most sites should do
- Allow Googlebot. It’s Search.
- Decide on Google-Extended the way you decide on other AI training crawlers. Blocking it has no Search cost; see should you block AI crawlers?.
- Verify before trusting. Treat a “Googlebot” that fails the check as a scraper; see fake AI bots.
Create a free account to see verified Google crawls next to fakes, and the people Gemini sends you, on one dashboard.
Check your own website
See which AI bots really visit your site — real or fake, page by page — and how many people AI assistants send you. Free.