Google Will Honor a Shared AI Training Opt-Out in 2027; 128 of 531 Top Sites Block AI Bots

Ras Al Khaimah, United Arab Emirates, 30 October 2026.

Google said on 30 October 2026 that it plans to support a shared, cross-company AI training opt-out in robots.txt, based on an Internet-Draft submitted the same day to the IETF's AI Preferences (AIPREF) working group. In a Search Central blog post by John Mueller, Google said a site that opts out of training with the new shared "policy token", and does not name Google-Extended, will be treated "as if the publisher had disallowed Google-Extended". Microsoft Bing announced it will honour the same signal from 1 February 2027, and the draft's author, Anthropic's Kevin Kelley, said Anthropic has already started implementing it.

To see how much work the change could save, we fetched the robots.txt files of the 1,000 most popular domains on the Tranco list. Of the 531 that returned a usable file, 128 (24.1%) already block at least one AI crawler or training token across the whole site, and 80 of those list five or more separate company tokens. None uses the new token yet.

What changed

• A new shared token. The draft, draft-kelley-aipref-training-preference-signal-00, defines one robots.txt user-agent value, __usage_ai-training. A normal group under it (Disallow and Allow lines) states a preference against or for AI training on those paths. It adds no new directives or grammar.
• Google will map it to Google-Extended. If a site disallows training with the shared token and has no Google-Extended group, Google plans to read that as a Google-Extended disallow. If a Google-Extended group exists, that company-specific rule still applies. Google's blog post says it aims to support the control in 2027, and a note added the same day to Google's common crawlers documentation says early 2027.
• Bing goes further. Microsoft's Bing blog post adopts the same robots.txt training token and adds a separate, page-level AI Grounding preference, with Allow, Disallow and a new DisallowExtended option that limits grounding to Microsoft's own products.
• Wildcard rules do not count. Under the draft, a User-Agent: * group never matches the policy token. A file that does not name the token expresses no training preference at all.

The draft is an individual submission. Its datatracker page notes that it is not endorsed by the IETF and has no formal standing in the standards process yet; it expires on 3 May 2027 unless updated.

How the signal is written

The draft's own example blocks training everywhere except one folder. It is an ordinary robots.txt group, so parsers that do not know the token should skip it the way they skip any unknown crawler:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft keeps this separate from crawl control. It says the token does not direct crawling, so Googlebot, Bingbot and other crawlers keep following the groups written for their own names. Bing's guide adds page-level versions for its own crawler (a Bingbot robots meta tag or X-Robots-Tag header with ai-training:disallow or ai-grounding:disallow), which override the robots.txt training setting for that page.

Our robots.txt check: where sites stand today

Sites that want to keep their content out of AI training currently write one group per company. The 128 sites that block at least one AI token site-wide name a median of 33 user-agent lines in their files, and the most common pattern was seven AI tokens blocked at once.

The per-company counts show how uneven those lists are. More sites block OpenAI's GPTBot (87) than Google-Extended (76), and 34 sites block GPTBot, ClaudeBot or CCBot without naming Google-Extended at all. If those sites switch to the shared token, Google's mapping would extend their opt-out to Gemini training too.

Token in robots.txt	Files naming it	Files blocking it site-wide
Bytespider	108	93
CCBot	115	90
GPTBot	133	87
ClaudeBot	129	86
Google-Extended	116	76
Meta-ExternalAgent	89	71
Applebot-Extended	86	63
PerplexityBot	118	63
anthropic-ai	68	46
__usage_ai-training	0	0

Source: SaaS Link Building crawl of 531 robots.txt files from the Tranco top 1,000, 30 October 2026. "Site-wide" means the homepage path is disallowed for that token under longest-match rules.

Two more numbers matter for the transition. Of the 116 files that name Google-Extended, 40 do not block it site-wide: 24 block only some paths and 16 allow it or leave the group empty. Under Google's stated plan, those explicit groups would keep deciding Google's behaviour even if the same file later adds a blanket shared opt-out. Separately, 19 files contain Content-Signal lines, a different syntax that the new draft does not cover.

Context: why this is different from earlier opt-outs

Google launched Google-Extended in 2023. Its documentation says the token controls both training of future Gemini models and grounding in Gemini Apps and Vertex AI, and does not affect Google Search inclusion or ranking. Our earlier study of AI crawler blocking among 465 top sites found the same company-by-company pattern for Meta's crawlers, and a separate 2,000-domain robots.txt crawl found many files carrying fields Google ignores. The new token avoids that problem by reusing the user-agent syntax every compliant parser already reads.

The approach also differs from Google's existing generative AI control for Search, which we covered when Google listed its AI Overviews and AI Mode opt-out in its core blocking guide. That control is about how Search features use content; the shared token is limited to model training as defined by the IETF vocabulary draft, which excludes models used only for ranking or classification.

Google's post ties the proposal to the EU Code of Practice for General Purpose AI model providers, which recognises robots.txt as a primary way for publishers to express machine-readable preferences. Search Engine Roundtable reported that OpenAI has also announced plans to adopt the signal; OpenAI's crawler documentation, which we checked, does not yet mention the token.

What this means for SEO teams

• Nothing changes today. Google and Bing will not read the token until 2027. Keep existing Google-Extended, GPTBot and ClaudeBot groups in place.
• Watch the Google-Extended trade-off. Google's post says sites that explicitly allow Google-Extended are eligible to appear and earn clicks through grounding in Gemini Apps. Our reading (Google has not spelled this out) is that a shared training opt-out, mapped to a Google-Extended disallow, would also remove that grounding eligibility unless the site adds an explicit Google-Extended allow group.
• Plan Bing separately. Sites using NoArchive or NoCache to limit Bing's AI use need to publish the new training and grounding preferences by 1 March 2027, after which Bing says those tags will no longer be read as AI preferences.
• Audit partial Google-Extended groups. If your file names Google-Extended with path rules, decide whether that group or a future shared opt-out should win, because Google will follow the specific group.
• Test before adding. Run any new group through a robots.txt tester and confirm it does not change rules for Googlebot or Bingbot, since a misplaced line can merge groups.

Methodology

On 30 October 2026 we took the top 1,000 domains from the daily Tranco list (list ID PYK6J) and requested https://<domain>/robots.txt, then the www host if the first request failed. We kept 531 responses that returned HTTP 200 with a non-HTML body. We parsed groups under RFC 9309 rules (consecutive user-agent lines share a group; longest matching path wins, Allow wins ties) and recorded, for ten tokens, whether a file names the token and whether the root path is disallowed for it. Wildcard groups were not counted as blocking any named token. The other 469 domains returned no usable file and are excluded. The counts describe the files on that date only.

Conclusion

Google, Microsoft and Anthropic have now pointed at the same robots.txt line for AI training preferences, and Bing has put dates on it. The signal is still an individual IETF draft, and Google's timing is only 2027. In our crawl, top sites currently keep lists of up to nine company tokens, and almost a quarter of them block at least one.

The practical risk is in the details. Google will defer to an existing Google-Extended group, and on our reading its mapping ties the training opt-out to Gemini grounding. Bing, by contrast, separates training from grounding. Teams that block AI training should map each company's rules before adding the shared line next year.

Questions and answers

What is the __usage_ai-training token?

It is a robots.txt user-agent value defined in the Internet-Draft draft-kelley-aipref-training-preference-signal-00, submitted on 30 October 2026. A Disallow or Allow group under it states a preference against or for AI training on those paths, for every provider that supports it.

When will Google start honouring it?

Google's Search Central blog says it aims to support the control in 2027, and its common crawlers documentation says early 2027. Until then, Google-Extended remains the way to opt out of Gemini training.

Does the shared token override Google-Extended?

No. Google says it will read a shared training opt-out as a Google-Extended disallow only when the site does not name Google-Extended explicitly. An existing Google-Extended group keeps applying.

Does blocking AI training affect Google Search or Bing rankings?

Google's documentation says Google-Extended does not affect inclusion or ranking in Google Search. Microsoft says its new training and grounding preferences do not change eligibility for Bing search results.

What deadline does Bing set?

Bing will honour the new preferences as it crawls from 1 February 2027. Sites relying on NoArchive or NoCache should publish the new training and grounding preferences by 1 March 2027, after which Bing will no longer read those tags as AI preferences.

###