DMGT Joins SPUR; 28 of 43 Member Sites Block Google-Extended, None Block Googlebot

DMGT, owner of the Daily Mail, has joined the board of the SPUR publisher coalition. Our crawl of 43 member robots.txt files found 28 block Google-Extended and 31 block GPTBot, yet none block Googlebot, the crawler behind Google's AI Overviews and AI Mode, and AI search bots are blocked far less often than training bots.

Vibhu Dhariwal
Cofounder & CTO, Digital Gratified
Published
Section
Studies
Reading time
8 minutes
Data card: 28 of 43 SPUR member sites block Google-Extended in robots.txt, while none block Googlebot

DMGT, the owner of the Daily Mail, Metro, The i Paper and New Scientist, has joined the board of the SPUR Coalition, the publisher group building standards for how AI companies use news content, Press Gazette reported on 17 November. To see what SPUR's members already tell AI crawlers today, we fetched the live robots.txt files of 49 member news sites on 20 November and parsed the rules for 19 crawlers. Of the 43 files we could read, 28 block Google-Extended, Google's token for Gemini training and grounding. None block Googlebot, the crawler that also feeds AI Overviews and AI Mode.

Key findings

  • 28 of 43 SPUR member sites (65%) block Google-Extended from article URLs; 0 of 43 block Googlebot, and only 1 blocks Bingbot.
  • Training crawlers are blocked more often than search crawlers: ClaudeBot by 33 sites and CCBot by 33, against 20 for Claude-SearchBot.
  • 8 of the 31 sites that block OpenAI's training crawler still let OAI-SearchBot in, and 11 of the 33 that block ClaudeBot still admit Claude-SearchBot.
  • 17 sites block all seven of the main training and model crawlers we tracked; 8 sites block none of the 17 AI tokens.
  • 6 sites already declare the new shared AI training opt-out token in robots.txt, before Google starts reading it.
  • Three DMGT titles we could read (dailymail.co.uk, metro.co.uk, inews.co.uk) block the same eight AI crawlers and leave Googlebot and Bingbot open.

What SPUR is and why DMGT matters

SPUR (Standards for Publisher Usage Rights) began as a British coalition and expanded this year into Europe and North America. Its 13 October announcement said membership had passed 80 organisations in more than 20 countries after the New York Times, AFP, The Economist and The Atlantic joined. The founding members listed there are the BBC, Financial Times, Guardian Media Group, Mediahuis, Sky News, Telegraph Media Group, the Associated Press and CMA Media.

The coalition's main technical output so far is the Content Telemetry Standard 1.0, published on 2 October. It defines how AI agents should report back to publishers when their content is retrieved, used for grounding, cited, presented or engaged with. It does not replace robots.txt: it describes usage reporting for licensed or permitted access, not crawl permission. That is why the robots.txt files of members are the clearest public record of what each publisher currently allows.

DMGT's Owen Meredith told Press Gazette the threat from AI scraping is one "no news organisation can combat alone". The robots.txt data shows how differently members act on their own today.

Training crawlers versus search crawlers

The biggest pattern in the files is a split between crawlers that collect training data and crawlers that power AI search answers. Several AI companies now separate these. OpenAI's crawler documentation says GPTBot covers model training while OAI-SearchBot decides whether a site can appear in ChatGPT search, and that each setting is independent. Anthropic's help page draws the same line between ClaudeBot and Claude-SearchBot.

Dot plot comparing how many of 43 SPUR member sites block training crawlers versus search crawlers: GPTBot 31 vs OAI-SearchBot 21, ClaudeBot 33 vs Claude-SearchBot 20, Google-Extended 28 vs Googlebot 0
Source: SaaS Link Building analysis of 43 SPUR member robots.txt files fetched on 20 November 2026. A crawler counts as blocked when its rules disallow all three test article paths.

Google is the outlier because it does not offer a separate search token for its AI features. Its common crawlers page says Google-Extended governs Gemini training and grounding in Gemini Apps and Vertex AI, and does not affect a site's inclusion in Google Search. Google's AI features guide says robots.txt rules for Googlebot are the crawl control for Search, including its AI features, and points site owners to nosnippet, max-snippet or noindex to limit what is shown. Separately, Google lists a Search generative AI control in Search Console for managing inclusion in AI Overviews and AI Mode, which we covered when it was added to Google's core blocking guide.

So the 28 SPUR sites that block Google-Extended have opted out of Gemini training and grounding, but every one of the 43 still lets Googlebot crawl, which keeps their articles available to AI Overviews and AI Mode unless they use a page-level or Search Console control that robots.txt cannot show.

Crawler by crawler

CrawlerOperatorBlockedPartly blockedNamed in file
ClaudeBotAnthropic33036
CCBotCommon Crawl33034
GPTBotOpenAI31033
BytespiderByteDance29130
Google-ExtendedGoogle28131
Applebot-ExtendedApple28129
PerplexityBotPerplexity28233
meta-externalagentMeta27027
ChatGPT-UserOpenAI23230
AmazonbotAmazon23126
OAI-SearchBotOpenAI21226
Claude-SearchBotAnthropic20227
Google-CloudVertexBotGoogle17016
BingbotMicrosoft105
GooglebotGoogle005

"Named in file" counts sites with a user-agent line for that crawler. A site can be blocked without naming a crawler (a catch-all rule), which is why Google-CloudVertexBot shows 17 blocks but 16 named groups. ChatGPT-User fetches pages when a person asks ChatGPT a question, and OpenAI says robots.txt rules may not apply to those user-initiated visits.

The one Bingbot block belongs to Mediahuis's independent.ie, which carries a User-agent: Bingbot group with Disallow: /. That rule shuts Microsoft's main search crawler out of the whole site, a step no other member takes.

Where the coalition disagrees with itself

Eight member sites (globalnews.ca, thelogic.co, moneysavingexpert.com, rusi.org, quantamagazine.org, blick.ch, timeshighereducation.com and tvo.org) block none of the 17 AI tokens we checked. Some name AI crawlers without shutting them out: moneysavingexpert.com names seven of the AI tokens we tracked but leaves article paths open, and which.co.uk blocks 11 AI user agents only from specific review and campaign URLs. At the other end, 17 sites, including bbc.co.uk, nytimes.com, dailymail.co.uk, theglobeandmail.com and hs.fi, block all of GPTBot, Google-Extended, ClaudeBot, CCBot, PerplexityBot, meta-externalagent and Applebot-Extended.

Founding members also differ. ft.com and theguardian.com allow GPTBot, and theguardian.com allows Google-Extended, while bbc.co.uk and news.sky.com block both. That spread is one reason SPUR is working on shared standards: individual robots.txt files cannot express licensing terms, only yes or no per crawler. We found a similar gap last month when 47 of 56 news sites' AI bot blocks missed Brave's crawler.

What this means for SEO teams

  • Separate training from search. If the goal is to stay visible in ChatGPT, Claude or Perplexity answers while declining model training, block GPTBot and ClaudeBot but leave OAI-SearchBot and Claude-SearchBot open. 8 SPUR sites already do this for OpenAI.
  • Don't expect Google-Extended to remove you from AI Overviews. Google's documentation says it does not affect Search. Use nosnippet, max-snippet or the Search Console control if you need to limit AI features in Search, and weigh the effect on regular snippets first.
  • Audit catch-all rules. A User-agent: * group can block or admit new crawlers you have never named. Check what your file does to tokens such as Google-CloudVertexBot and Claude-SearchBot.
  • Watch for the shared signal. Google says it intends to support the AIPREF training preference (user-agent: __usage_ai-training) in early 2027, which we covered in our report on Google's plan to honor a shared AI training opt-out. 6 of the 43 SPUR files already carry User-agent: __usage_ai-training with Disallow: /: amlintelligence.com, capital.bg, hongkongfp.com, observer.co.uk, radiofrance.fr and stuff.co.nz.

For a wider benchmark beyond news publishers, see our earlier study of AI crawler blocking across top sites.

Methodology

We built a list of 49 news and media sites from members named on SPUR's 13 October announcement and its members page, taking one flagship domain per member publisher and excluding trade associations without a news site. We added three more DMGT titles (metro.co.uk, inews.co.uk, newscientist.com) as a separate check. On 20 November 2026 at about 01:20 UTC we requested each site's /robots.txt over HTTPS with a crawler user agent that identified itself, following redirects. Six member files could not be read: afp.com, volkskrant.nl, lequipe.fr, zonebourse.com and nationalpost.com returned HTTP 403, and itv.com did not connect. newscientist.com returned HTTP 406. We did not retry with a disguised user agent, leaving 43 member files and 2 extra DMGT files.

For each of 19 crawler tokens we selected the user-agent group naming that token (case-insensitive), or the * group if none did, and applied Google's longest-match rule with Allow winning ties. We tested three paths: /, /news/sample-article-2026 and /2026/11/20/sample-story/. A crawler is "blocked" when all three are disallowed, "partly blocked" when some are, and allowed otherwise. Robots.txt shows stated crawl permissions only; it does not show licensing deals, server-side bot blocking, page-level meta tags or Search Console settings.

Conclusion

SPUR's members broadly reject AI training crawlers, with two thirds blocking Google-Extended and more than three quarters blocking ClaudeBot and CCBot. But they are far more willing to let AI search crawlers in, and not one of them blocks Googlebot. For Google's AI features, that means the coalition's robots.txt files change little: the content Google uses for AI Overviews and AI Mode still comes through the same crawler as regular search.

DMGT's arrival adds the Daily Mail's weight to the push for licensing standards. Until those standards reach the crawlers themselves, robots.txt remains a blunt yes or no, and the members' files show they are using it in very different ways.

Questions and answers

What is the SPUR Coalition?

SPUR (Standards for Publisher Usage Rights) is a publisher coalition building industry standards for how AI companies use news content. It said in October 2026 that it had passed 80 members in more than 20 countries, and DMGT, owner of the Daily Mail, joined its board in November 2026.

Does blocking Google-Extended remove a site from AI Overviews?

No. Google's crawler documentation says Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI and does not affect a site's inclusion in Google Search. Google says Googlebot rules, snippet controls and the Search generative AI control in Search Console manage AI features in Search.

How many SPUR member sites block Googlebot?

None. Of the 43 SPUR member robots.txt files we parsed on 20 November 2026, 0 blocked Googlebot from article URLs, while 28 blocked Google-Extended and 1 (independent.ie) blocked Bingbot.

Can a site block AI training but stay in ChatGPT search?

Yes. OpenAI says GPTBot (training) and OAI-SearchBot (ChatGPT search) are controlled independently. In our crawl, 8 of the 31 SPUR sites that block GPTBot still allow OAI-SearchBot.

Do any publishers use the __usage_ai-training token yet?

Yes. 6 of the 43 SPUR member files we read already include user-agent: __usage_ai-training with Disallow: /. Google says it intends to support this shared training preference signal in early 2027.

More releases

  1. Updates

    Google Adds an AI Mode Sidebar to Its Homepage; Search Button Held in 30 of 30 Checks

    Google began rolling out a collapsible sidebar of recent AI searches on its desktop homepage on 19 November 2026, Robby Stein said. Two days later, all 30 signed-out sessions we loaded across 20 locales still showed the classic Google Search button, with no sidebar or AI Mode button.

  2. Studies

    Google Calls '7 Tips' Posts Commodity Content; They Held 39% of Page One in Our Test

    Google is telling brands to stop publishing generic "Top 10" tips posts, but its results still reward them. Across 30 US tips searches on 21 November, numbered lists held 90 of 233 page-one organic results and a third of AI Overview citations, and a page titled exactly like Google's own commodity example ranked first.

  3. Industry news

    Google Adds Target, Experian and H&R Block to Search Apps; Help Page Lists 0 of 8

    Google is adding Target, Albertsons, Safeway, Shaw's, Experian and H&R Block as Connected Apps in Search and the Gemini app, with Angi and Thumbtack to follow. Its Search help page lists none of the eight yet, and in our 16-search test a partner's own site ranked in the organic top 10 on 10.

Every release

Tell us the site and the goals.

Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.