Google Details How to Allow Google-Agent; 24 Top Sites Block a Name Google Never Listed
Google published a guide on 17 November explaining how to let Google-Agent through CDNs, firewalls and bot managers. Our crawl of 1,159 top-site robots.txt files the same day found 24 naming an undocumented GoogleAgent-Mariner token, against 14 naming Google-Agent, which Google says generally ignores robots.txt.

Ras Al Khaimah, United Arab Emirates, .
Google on 17 November published a new crawling guide, Allow Google-Agent traffic, telling site owners how to stop content delivery networks (CDNs), web application firewalls (WAFs) and bot-detection services from blocking the agent Google uses to browse and act on websites for its users. The crawling documentation changelog describes it as a technical guide for publishers that want to enable Google-Agent. Search Engine Roundtable reported the page the same morning.
The guide is about firewall and bot-manager rules, not robots.txt. That matters, because our crawl of 1,159 robots.txt files from the 2,000 most popular domains, run a few hours after the page went live, found that top sites aiming rules at Google's agents mostly name tokens Google does not document. 24 files name "GoogleAgent-Mariner" and 16 name "GoogleAgent-Shopping", against 14 that name Google-Agent itself, a fetcher Google says generally ignores robots.txt anyway.
What changed
Google-Agent is not new. Google added the user agent on 20 March 2026, together with its own IP range file, and documented an experimental signing scheme on 4 May. What the new page adds is an operating manual for allowing that traffic through security layers. Its main steps:
- Match three signals. Confirm the User-Agent header contains Google-Agent, confirm the IP is in user-triggered-agents.json, and, where present, verify the request signature against the key set Google publishes for the agent.bot.goog identity.
- Skip challenges for verified requests. Google says verified requests should bypass bot challenges, including interactive CAPTCHAs and JavaScript challenges, not just HTTP blocks.
- Cover the whole journey. Allow rules should reach on-site search, filters, detail pages, add-to-cart, booking and the XHR, GraphQL or api.* calls behind them, not only homepage loads.
- Fix rule order and rate limits. The allow rule must be evaluated before broad data-centre ASN blocks, managed bot rules or category-level AI agent blocks, and per-IP rate limits should account for the fact that many users' agent sessions share the same egress IPs.
- Keep headers intact. Load balancers and proxies must pass on the original client IP, the User-Agent and the Signature, Signature-Input and Signature-Agent headers.
The guide frames all of this as keeping "existing security controls" in place while letting legitimate agent requests through.
| Date | Google-Agent documentation milestone | Days after first listing |
|---|---|---|
| 20 March 2026 | User agent and user-triggered-agents.json IP ranges added | 0 |
| 4 May 2026 | Web Bot Auth verification guide added (experimental) | 45 |
| 17 November 2026 | Allow Google-Agent traffic guide published | 242 |
Our check of the live files
We pulled Google's agent files a few minutes after the guide appeared. The IP file was regenerated on 16 November and still lists the same 20 prefixes (12 IPv4, 8 IPv6) we counted in our earlier guide to spotting Google-Agent in server logs, covering 70,016 IPv4 addresses. None of those prefixes overlap Google's four other crawler and fetcher files, so a firewall allowlist built from Googlebot's ranges will not cover the agent.
The agent.bot.goog key directory served five Ed25519 public keys with a one-hour cache lifetime. The directory response was itself signed, with a signature valid for 300 seconds, which lets a verifier confirm the key list came from Google before trusting it. That follows the HTTP Message Signatures standard (RFC 9421) the guide cites.
Key findings: what top sites put in robots.txt
Four findings stand out.
- Undocumented agent names outnumber the real one. 36 files name at least one agent-style Google token. GoogleAgent-Mariner appears in 24 and GoogleAgent-Shopping in 16. Neither name appears in any of the Google crawler pages we checked, and Google-Agent's documented user agent strings carry only the Google-Agent token.
- Amazon drives most of it. 15 of the 24 Mariner files and 15 of the 16 Shopping files belong to Amazon storefront and video domains, which disallow both names site-wide. None of those Amazon files names Google-Agent.
- Google-Agent rules are rare and mostly symbolic. 14 files name Google-Agent, and 9 of those groups include Disallow: /. Three files (USA Today, Le Monde and The Conversation) name both Google-Agent and the Mariner token. Google's list of user-triggered fetchers says these fetchers generally ignore robots.txt rules because a user asked for the fetch, so those lines may have no effect.
- Training controls dominate. Google-Extended, the documented token for Gemini training and grounding, appears in 243 files, 166 of them with Disallow: /. Googlebot is named in 146 files, but only 6 of those groups include Disallow: /.
The pattern suggests many top sites are copying bot names from third-party block lists rather than from Google's documentation. For training and grounding, robots.txt works through Google-Extended. For an agent acting on a user's request, Google's own answer is the firewall layer covered in the new guide.
What this means for SEO teams
- Decide policy at the firewall, not in robots.txt. If you want agents to complete bookings, sign-ups or purchases, follow the new guide and allow verified Google-Agent traffic in your CDN or WAF. If you do not, a robots.txt line will not reliably stop it, by Google's own description of user-triggered fetchers.
- Audit robots.txt for names Google never published. Lines for GoogleAgent-Mariner, GoogleAgent-Shopping, GoogleAgent-URLContext or Gemini-Deep-Research do not match any token on Google's crawler pages. They are harmless but give a false sense of control. Our earlier study of AI crawler blocking showed how unevenly top sites target each company's bots.
- Put the agent rule above managed bot rules. Google's troubleshooting list puts rule priority first. A verified allow rule that sits below an "AI agents" category block or a data-centre ASN block will never fire.
- Test the deep pages. Google calls out on-site search, filters, add-to-cart and API endpoints. These can sit behind separate rate limits or CAPTCHA widgets that a homepage check will miss.
- Refresh the IP file and keys automatically. Google regenerates its IP files regularly, and the key directory sets a one-hour cache. Static allowlists drift. The same discipline applies to the Googlebot ranges we covered in our report on CDN moves and crawl spikes.
Methodology
On 17 November 2026, between 13:20 and 13:30 UTC, we requested /robots.txt from each of the top 2,000 domains in the Tranco list (ID K97GW), trying the bare domain and then the www host, with a user agent identifying the request as a robots.txt survey. We kept 1,159 responses that returned HTTP 200 with a non-HTML body. The rest returned errors, 4xx or 5xx codes (including 403 from 50 of the top 1,000), or are infrastructure domains that serve no website. We parsed each file into user-agent groups, counting consecutive User-agent lines as one group, and recorded a file as naming a token when any group listed it (case-insensitive), and as blocking it when that group included Disallow: /. A group can include Allow lines that narrow a Disallow: /, so the blocking count is an upper bound. We searched seven Google crawler documentation pages fetched the same day (crawler overview, common crawlers, special-case crawlers, user-triggered fetchers, request verification, Web Bot Auth and the new Google-Agent guide) for each token. We downloaded all five of Google's crawler IP files, compared prefixes with Python's ipaddress module, and fetched the agent.bot.goog key directory with its response headers.
Conclusion
Google's new guide moves the Google-Agent conversation from identification to access. Eight months after the agent first appeared in its documentation, Google is telling site owners exactly where their security stack is likely to block it: rule order, interactive challenges, deep-page APIs and shared-IP rate limits.
Top sites are not yet working at that layer. In our crawl, the most common robots.txt rules aimed at Google's agents used names Google has never published, and the documented Google-Agent token appeared in just 14 of 1,159 files. Teams that care about agent traffic, in either direction, should look at their CDN and firewall settings first.
Questions and answers
What did Google publish about Google-Agent on 17 November 2026?
Google added a page called Allow Google-Agent traffic to its crawling documentation. It explains how to verify Google-Agent requests by user agent, IP range and optional Web Bot Auth signature, how to allow them in a CDN, firewall or bot-detection service, and how to troubleshoot rule priority, CAPTCHAs, API endpoints, shared-IP rate limits and proxy headers.
Can robots.txt block Google-Agent?
Not reliably. Google lists Google-Agent among its user-triggered fetchers and says these fetchers generally ignore robots.txt rules because a user requested the fetch. Google's new guide deals with access at the CDN, firewall and bot-detection layer instead.
Is GoogleAgent-Mariner an official Google user agent?
It does not appear on any of the seven Google crawler documentation pages we checked on 17 November 2026, including the crawler overview and the list of user-triggered fetchers. Google-Agent's documented user agent strings contain only the Google-Agent token. Even so, 24 of the 1,159 top-site robots.txt files we crawled named GoogleAgent-Mariner.
How do I verify that a request comes from Google-Agent?
Google says to confirm the User-Agent header contains Google-Agent, confirm the IP address is in user-triggered-agents.json, and, if the request is signed, verify the Signature and Signature-Input headers for Signature-Agent g="https://agent.bot.goog" under Web Bot Auth. Verified requests should then skip bot challenges.
More releases
-
Studies
Meta Paused Its Crawling Terms; Only 2 of 485 Top Sites Block Its Ads Crawler
Meta has paused the business terms that would have let it crawl advertisers' websites from 20 November. Our repeat robots.txt crawl of the top 1,000 domains finds little change: 12.8% of 485 sites now block Meta's AI training crawler, up from 12.0% in October, and only 2 block Meta-ExternalAds.
-
Studies
Are AI Overviews Cutting Clicks More Than Google Admits? The Evidence Says Likely
A pre-registered experiment with 1,065 US users found AI Overviews cut organic clicks by about 38% on the searches where they appear, while Google speaks only of stable totals and higher quality clicks. Our pull of 24 mobile results pages found the AI Overview on top of all 22 that showed one.
-
Explainers
Ask Maps Will Phone Local Businesses; 52 of 80 Listings We Pulled Were Phone-Only
Google's Ask Maps can now find beauty, pet care, auto and home repair appointments in the US and call local businesses for the user. In our pull of 80 Google Maps listings across those four categories, 52 offered no online booking link, only a phone number. Google's help pages explain how owners can opt out.
Tell us the site and the goals.
Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.