Here is the short answer: Googlebot is the crawler that puts your pages into Google Search, and blocking it removes you from Search entirely, including AI Overviews and AI Mode. Google-Extended is not a crawler at all. It is a permission token that tells Google whether your content may be used to train future Gemini models and to ground answers in the Gemini app, and blocking it does not touch your Search ranking or your place in AI Overviews. Gemini, AI Overviews, and AI Mode all work from the regular Search index, which Googlebot builds. So the decision most site owners actually face is a narrow one: leave Googlebot alone, and decide separately whether to block Google-Extended. Get those two the wrong way round and you either vanish from Google or give away training rights you meant to keep.
The Google Crawler Family at a Glance
Google does not send one bot to your site. It sends a family of crawlers, each with its own name, its own job, and its own robots.txt token. Google publishes the full list in its official documentation on common crawlers, and it is the only source you should trust for what each one does. The names look similar, which is exactly why people mix them up, so here is the family in plain language before we go into detail.
Googlebot is the main crawler. It fetches pages for Google Search, including Discover, Images, Video, and News. Googlebot-Image, Googlebot-Video, and Googlebot-News are specialised members of the same family that handle images, video, and news content. GoogleOther is a generic crawler that various Google product teams use to fetch publicly accessible content, with GoogleOther-Image and GoogleOther-Video as its image and video variants. Google-Extended is the odd one out: it has no crawler, no visits, and no user agent string of its own. It is a token you set in robots.txt. There are also smaller names you will see in logs, such as AdsBot for checking landing page quality, Storebot for Shopping, Mediapartners-Google for AdSense, and Google-InspectionTool, which is the crawler behind Search Console testing tools such as the URL inspection tool and the Rich Results Test. All of Google common crawlers obey robots.txt rules when they crawl automatically, according to Google own documentation.
The key idea to carry through this article is that each token controls one job. A rule aimed at Googlebot affects Search. A rule aimed at Google-Extended affects Gemini training and grounding. A rule aimed at Googlebot-Image affects images. They do not spill over into each other, and Google documents exactly which product each token affects. That precision is your friend, because it means you can be selective instead of guessing.
Googlebot: The One You Almost Never Block
Googlebot is the program Google uses to discover and download web pages so they can be added to the Google index. When you publish a blog post, Googlebot is the visitor that reads it, follows its links, renders its JavaScript, and hands it to the indexing system. Without a Googlebot visit, a page does not enter the index, and without the index there is no Google Search listing, no Discover appearance, and no AI Overview citation. Everything downstream depends on this one crawler doing its job. Google's own Googlebot documentation describes both the smartphone and desktop crawlers and how they access your site.
Googlebot is not a single user agent string. The family includes a smartphone crawler and a desktop crawler, both using the Googlebot token. Since Google completed the move to mobile-first indexing, the smartphone crawler does almost all of the work: Google indexes the mobile version of your pages by default. The desktop crawler still visits, but it is the minority. Both render JavaScript, which matters if your site loads content with scripts. Google processes JavaScript pages in three phases, crawling, rendering, and indexing, so a page that only appears after scripts run can still be indexed, as long as Googlebot can fetch and render it.
Googlebot reads robots.txt before each crawl and respects noindex directives. It also throttles itself when your server slows down, and it identifies itself clearly, so you can verify a visit with a reverse DNS lookup. The user agent string looks like Googlebot/2.1, and Google publishes the IP ranges its crawlers use, refreshing them daily. If a visitor claims to be Googlebot but its reverse DNS does not resolve to a googlebot.com hostname, it is not Googlebot. Our guide to verifying AI bots with reverse DNS walks through that check.
Blocking Googlebot is almost never what you want. A robots.txt group that disallows Googlebot from your whole site tells Google to stop crawling, and pages that are already indexed will gradually drop out of Search. There is no separate switch that keeps your pages in Search but hides them from AI features built on Search, because those features read the same index. If your pages are not in the index, they cannot be cited in an AI Overview, in AI Mode, or in a regular result. That is the whole story, and it is why every reputable guide starts with the same warning: leave Googlebot alone.
Googlebot-Image, Googlebot-Video, and Googlebot-News
The specialised Googlebot variants exist because different kinds of content need different handling. Googlebot-Image fetches images for Google Images and for every Search feature where images, logos, and favicons appear, including Discover and Video. Its full user agent string is Googlebot-Image/1.0, and its robots.txt tokens are Googlebot-Image and Googlebot. That second token matters: a rule aimed at Googlebot also covers Googlebot-Image, because Google treats the plain Googlebot token as matching the whole family.
Googlebot-Video works the same way for video files, with the token Googlebot-Video and the user agent string Googlebot-Video/1.0. Googlebot-News is the odd one: it has no separate HTTP user agent string of its own, and its crawling is done with various Googlebot user agent strings. Its token is Googlebot-News, and rules aimed at it affect the Google News product, including news.google.com and the Google News app.
For most site owners these three need no separate rules. If you allow Googlebot, your images, videos, and news pages are covered. The only common reason to write a rule for one of them is to stop Google from indexing your images while keeping your text pages in Search, for example if you are a photographer who wants text found but images kept out of Google Images. In that case a group naming Googlebot-Image with Disallow for your image folders does the job without touching your articles. The reverse is also true: blocking Googlebot-Image alone does not remove your pages from Search, it only removes the images. Keep the tokens separate in your head, because Google keeps them separate in its system.
Google-Extended: A Token, Not a Crawler
Google-Extended is the most misunderstood name in this whole family, and the misunderstanding costs site owners in both directions. Some block it thinking they have opted out of AI Overviews, and they have not. Others allow it thinking it is a search crawler, and they have given away training rights they never meant to give. So here is the truth, straight from Google own documentation.
Google-Extended is a standalone product token. It has no HTTP user agent string of its own, and no crawler ever visits your site calling itself Google-Extended. Google announced it on 28 September 2023 as a way for publishers to choose whether their content helps improve the AI models behind what was then called Bard and the Vertex AI generative APIs. Today it controls two uses of content Google has already crawled: training future generations of Gemini models, which power the Gemini app and the Vertex AI API for Gemini, and grounding, which means handing content from the Search index to the model at the moment a user asks a question, in Gemini Apps and in Grounding with Google Search on Vertex AI.
The sentence that settles the visibility question is also from Google documentation: Google-Extended does not impact a site inclusion in Google Search, nor is it used as a ranking signal in Google Search. Blocking it changes nothing about your listings, your positions, or your organic clicks. It also does not remove you from AI Overviews or AI Mode, because those are Search features built on the regular index, and Google-Extended is explicitly not the control for them.
There is one real trade-off, and it is on the Gemini side. Google says that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. Blocking it therefore means your content will not be used to ground Gemini app answers, so you lose that potential source of clicks and citations inside the Gemini app. That is a genuine cost, and it is the reason the decision deserves thought rather than a reflex block. If your content is the kind of thing people ask Gemini about, and you want those readers, allowing Google-Extended is the price of admission. If your content is creative work, research, or reporting you do not want feeding a model, blocking it is the right call. Either way, make the decision knowing what it actually controls.
One more thing to note: Google announced on 30 October 2026 that it plans to support a shared AI training control in robots.txt, built on work from the AIPREF working group at the Internet Engineering Task Force, with support aimed for early 2027. The shared signal is a single robots.txt group naming a special token, and Google says it will treat an opt-out through that signal as if the publisher had disallowed Google-Extended, unless Google-Extended is named explicitly. Until that support is live, Google-Extended remains the control Google actually reads, so use it today and treat the shared signal as something to watch, not something to rely on yet.
GoogleOther, GoogleOther-Image, and GoogleOther-Video
GoogleOther is the generic crawler. Google documentation describes it as the crawler that may be used by various product teams for fetching publicly accessible content from sites, and rules aimed at its token do not affect any specific product. In practice that means GoogleOther visits are one-off fetches for internal research and development, or for product teams building something new, rather than for Search indexing.
Google added two variants in May 2024: GoogleOther-Image, which fetches publicly accessible image URLs, and GoogleOther-Video, which fetches publicly accessible video URLs. Their tokens are GoogleOther-Image and GoogleOther-Video, with GoogleOther as a second token for each, and their user agent strings are GoogleOther-Image/1.0 and GoogleOther-Video/1.0.
Should you block GoogleOther? For almost every site owner, no. It does not affect Search, it does not train Gemini, and blocking it gains you nothing while potentially breaking a Google product feature that uses your public content in a way you would have been fine with. The exception is a site with a strict policy against any non-Search fetching, such as a site under a licensing deal that forbids all secondary use. If that is you, a single group disallowing GoogleOther covers the whole family, including the image and video variants. For everyone else, leave it alone and spend your attention on the tokens that actually matter.
The Smaller Names in Your Logs
A few more Google crawlers show up in server logs often enough to be worth naming, so you are not surprised when you see them. AdsBot checks landing page quality for Google Ads, and its variants cover mobile and video ads. Storebot fetches product data for Google Shopping. Mediapartners-Google is the AdSense crawler: it visits your pages to decide which ads fit, and blocking it can break ad targeting on your own site, which is the opposite of what most publishers want. Google-InspectionTool is the crawler used by Search testing tools such as the Rich Results Test and the URL inspection tool in Search Console. Apart from its user agent and token, it mimics Googlebot, and it only visits when you or someone with access to your Search Console property asks it to test a page.
There is also Google-CloudVertexBot, which Google documents as crawling content for Vertex AI products. It is a real crawler with real visits, unlike Google-Extended, and it is controlled by its own token. If you see it in your logs and wonder what it is doing, the answer is that it is fetching pages for use in Vertex AI features, not for Search indexing. Whether to block it depends on the same question as Google-Extended: are you comfortable with your content being used inside Google AI products beyond Search? If not, a group naming it with a Disallow rule is the way to say so.
None of these smaller crawlers should be blocked casually. Each one exists because a Google product needs to fetch your pages to do a job you probably want done: serving relevant ads, listing your products, or testing your pages. Block them only when you have a specific reason tied to a specific product, not because an unfamiliar name appeared in a log file.
How Gemini, AI Overviews, and AI Mode Actually Get Your Content
This is the section that clears up the most confusion, so here it is plainly. AI Overviews and AI Mode are surfaces inside Google Search, not separate products with their own crawlers. They are built on the regular Search index, and that index is built by Googlebot. A page must be indexed and eligible to show a snippet before it can be cited in an AI Overview or an AI Mode answer. That is why blocking Google-Extended does nothing to AI Overviews: the token governs training and Gemini app grounding, and AI Overviews are neither.
Gemini, the standalone app, works differently. Its answers can be grounded in content from the Search index at the moment a user asks a question, and that grounding is one of the two uses Google-Extended controls. The other use is training future Gemini models. So Google-Extended is the Gemini control, Googlebot is the Search control, and they do not overlap.
If you want out of AI Overviews and AI Mode specifically, Google now offers a real control for that, and it lives in Search Console rather than in robots.txt. The Search generative AI control, under Settings in a Search Console property, lets you include or exclude your site links and content from AI Overviews, AI Mode, and generative AI features in Google Discover. Google rolled it out worldwide as of 31 August 2026, with Include as the default. Excluded sites do not receive traffic or impressions from those features, and Google says the control is not used as a ranking or inclusion signal for regular Search. It does not affect AI training, which is still Google-Extended job. The control is property-wide, not page-level, so it is an all-or-nothing choice for the whole site.
Before that control existed, the only levers were the page-level snippet controls: nosnippet, max-snippet, data-nosnippet, and noindex. Those still work, and Google AI features documentation still points to them. A nosnippet meta tag on a page removes its text snippet from all search results, including AI Overviews and AI Mode, and prevents the content from being used as a direct input for those features. The cost is that you also lose the snippet in regular results, which can hurt click-through rates. A max-snippet value limits how much text Google may show, which is a softer version of the same idea. Use these when you want to control specific high-value pages, and use the Search Console control when you want the whole site out of AI features. Do not use Google-Extended for this job. It is the wrong tool, and it will not work.
The Decision Table: Who Should Block What
Here is the decision in table form, because this is the part you will come back to. Read your row, then use the matching robots.txt recipe in the next section.
Most blogs, businesses, and publishers. Allow Googlebot, allow GoogleOther, and decide on Google-Extended based on how you feel about Gemini training. If you are unsure, blocking Google-Extended is the safer default for original content, because it costs you nothing in Search and keeps your work out of future Gemini training. Leave AI Overviews alone unless you have a specific reason to leave them, because they can send traffic.
News sites and publishers who live on Google News. Allow Googlebot and Googlebot-News without exception. Your Google News presence depends on them. Decide on Google-Extended separately, and remember that Google-Extended has no effect on your News listings either.
Photographers, illustrators, and visual artists. Allow Googlebot for your text pages, but consider a targeted Googlebot-Image rule for folders that hold your original work if you do not want it in Google Images. Block Google-Extended, because visual work is exactly the kind of content that ends up in training data. Our guide to protecting images from AI training goes deeper on this.
E-commerce and product sites. Allow Googlebot, Storebot, and Mediapartners-Google. Storebot feeds Shopping listings and Mediapartners-Google feeds AdSense, so blocking either one directly costs you money. Google-Extended is a branding and content decision for your product descriptions and guides, not a sales decision, so decide it on its own merits.
Sites that do not want any AI involvement. Block Google-Extended, block the training crawlers from other companies, and use the Search Console generative AI control to leave AI Overviews and AI Mode. Accept that the Search Console control costs you the traffic those features would have sent. Do not block Googlebot as part of this, unless you genuinely want to leave Google Search entirely, because that is what blocking Googlebot means.
Sites that want maximum AI visibility. Allow everything, including Google-Extended. Your content will be eligible for Gemini grounding, which Google says can bring clicks through the Gemini app, and for AI Overviews and AI Mode, which can bring clicks through Search. This is a legitimate strategy for content designed to be cited, such as reference guides and explainers. It is a bad strategy for creative work you want to keep yours.
The Exact robots.txt Rules for Each Choice
Here are three ready recipes. Pick one, paste it into your robots.txt file, and check it before you publish. If you build your file with our free robots.txt generator, you can tick the same choices and get the same output without typing the syntax by hand.
Recipe 1: Stay visible everywhere, opt out of Gemini training only. This is the most popular choice, and it is the one we recommend for most sites:
User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /
Googlebot, GoogleOther, and all the search crawlers are untouched, so Search, Discover, Images, Video, and News all keep working. The training crawlers are told to stay out. AI Overviews and AI Mode keep working too, because they run on Googlebot, and if that surprises you, re-read the section above: it is the most common misunderstanding in this whole topic.
Recipe 2: Stay in Search, leave AI Overviews and AI Mode as well. Keep Recipe 1, and add the Search Console step: open Search Console, open your property, go to Settings, then Search generative AI, and choose Exclude. Do not try to do this with robots.txt, because no robots.txt token controls AI Overviews. If you need it page by page instead of site-wide, add a nosnippet meta tag to the specific pages instead of using the Search Console control. The meta tag looks like this, placed in the head of the page:
<meta name="robots" content="nosnippet">
Remember the cost: nosnippet also removes the text snippet from regular search results for that page, so use it on pages where the snippet itself is the problem, not as a blanket policy.
Recipe 3: Maximum AI visibility. Do not add any Google rules at all. Allow Googlebot, allow Google-Extended, and leave the Search Console control on Include. Your content stays eligible for Search, for AI Overviews and AI Mode, and for Gemini grounding. This is the right choice for reference content you want cited everywhere, and the wrong choice for work you want to keep out of training data.
After pasting any recipe, verify it. Open yoursite.com/robots.txt and confirm the new lines appear. Then paste the file into our robots.txt tester and check each crawler one by one: Googlebot should show as allowed, Google-Extended as disallowed, and the rest as you intended. Robots.txt changes take time to be picked up, and Google may take days to re-crawl and apply a new rule, so do not expect instant results. Check again after a week, and check your Search Console crawl statistics to confirm Googlebot is still visiting at its normal rate.
Common Mistakes to Avoid
The first mistake is blocking Googlebot by accident. It usually happens with a blanket rule: a group naming every user agent with Disallow for the whole site. That rule tells Googlebot to stay away, and your pages start dropping out of Search. If your traffic falls off a cliff after a robots.txt change, check for a wildcard group first. It is the most common cause, and it is also the easiest to fix: delete the wildcard group and leave the named groups in place.
The second mistake is blocking Google-Extended and expecting to leave AI Overviews. As this article has said several times now, that does not work. AI Overviews use Googlebot. If you want out of them, use the Search Console control or nosnippet. Blocking Google-Extended and then wondering why your content still appears in an AI Overview is a support ticket we have seen many times, and the answer is always the same: you blocked the wrong token.
The third mistake is assuming a robots.txt rule removes content already collected. It does not. A disallow rule only stops future crawling. If Google already crawled your pages before you added the rule, the content is already in the index and may already have been used for training. The rule protects what you publish next, not what you published last year. That is true for every crawler and every robots.txt rule, and it is worth saying plainly so nobody expects a settings change to erase the past.
The fourth mistake is trusting a user agent string without verifying it. User agent strings can be faked, and scrapers do fake them. If a visitor claims to be Googlebot, verify it with a reverse DNS lookup before believing it. The check is simple: look up the IP address, confirm the hostname ends in googlebot.com, and confirm the forward lookup resolves back to the same IP. Our reverse DNS verification guide shows the exact steps. Never build a blocklist from log entries you have not verified, because you may end up blocking a real Google crawler or allowing a fake one.
Frequently Asked Questions
Will blocking Google-Extended hurt my Google ranking?
No. Google states in its crawler documentation that Google-Extended does not impact a site inclusion in Google Search and is not used as a ranking signal. Your listings, positions, and organic clicks should not move when you add the rule. If your ranking changes after adding it, something else changed at the same time.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are built from the regular Search index, which Googlebot crawls. Google-Extended only controls Gemini training and grounding in the Gemini app. To leave AI Overviews, use the Search generative AI control in Search Console, or add a nosnippet tag to specific pages.
Is Google-Extended a real crawler I can see in my logs?
No. Google-Extended has no crawler and no user agent string of its own. It never visits your site. It is a token Google reads from your robots.txt to decide how already-crawled content may be used. You will never see it in a server log, and that is normal.
What is the difference between Googlebot and GoogleOther?
Googlebot crawls for Google Search and its features, including Discover, Images, Video, and News. GoogleOther is a generic crawler used by various Google product teams to fetch publicly accessible content, and rules aimed at it do not affect any specific product. Block Googlebot and you leave Search. Block GoogleOther and almost nothing changes for most sites, so most site owners leave it alone.
Should I block Googlebot-Image to protect my photos?
Only if you want your images out of Google Images while keeping your text pages in Search. A group naming Googlebot-Image with Disallow for your image folders does that without touching your articles. Blocking the plain Googlebot token would also block the image crawler, but it would remove your pages from Search too, which is almost certainly not what you want.
How do I get out of Gemini answers but stay in Google Search?
Block Google-Extended in robots.txt. That opts your content out of Gemini training and out of grounding in the Gemini app, while leaving Googlebot untouched, so your Search listings and AI Overview eligibility stay exactly as they are. Check the rule in our robots.txt tester afterwards, and remember that the rule only affects future use, not content already collected.
The bottom line is a two-line summary you can pin above your desk. Googlebot builds your visibility in Google Search, including AI Overviews and AI Mode, so leave it alone. Google-Extended decides whether your content trains Gemini and grounds Gemini app answers, so block it or allow it based on what you want for your content. Everything else in the Google family is a detail, and now you know which details matter.
