12% of 465 Top Sites Block Meta's AI Training Crawler in Robots.txt, but None Block Its Ads Crawler

Our analysis of robots.txt files from the 1,000 most popular domains on the Tranco list found that 12.0% of 465 sites that admit crawlers by default block Meta's AI training crawler at the root, while 15.3% block GPTBot. None block Meta-ExternalAds, the agent behind Meta's ad products, days before Meta's new crawling terms take effect.

Vibhu Dhariwal
Cofounder & CTO, Digital Gratified
Published
Updated
Section
Studies
Reading time
4 minutes
Data card: 12.0% of 465 top sites block Meta's AI training crawler in robots.txt

Twelve days before Meta's updated Commercial Terms take effect on 30 October, SaaS Link Building analysed the robots.txt files of the 1,000 most popular domains on the Tranco research ranking to see how the web's largest sites treat AI crawlers. Of 465 sites that allow crawlers by default, 71 (15.3%) block OpenAI's GPTBot from the root, 56 (12.0%) block Meta-ExternalAgent and none block Meta-ExternalAds.

The numbers show that AI crawler policy at the top of the web is selective rather than wholesale, and that Meta's newer agents are rarely addressed by name.

Key findings

  • Training crawlers are blocked most. CCBot (17.2%), Bytespider (17.0%), GPTBot (15.3%) and ClaudeBot (15.1%) top the list.
  • Search crawlers are blocked about half as often. OAI-SearchBot is blocked by 6.0% of sites and Claude-SearchBot by 7.1%. 43 of the 71 GPTBot blockers still allow OAI-SearchBot, and 37 of the 70 ClaudeBot blockers still allow Claude-SearchBot.
  • Meta follows the same pattern. 37 of the 56 sites that block Meta-ExternalAgent still allow Meta-WebIndexer, which Meta says helps it cite and link content in Meta AI.
  • Meta-ExternalAds is almost invisible. No site blocks it at the root, and only 4 of the 465 robots.txt files mention it at all.
  • Blocking is not consistent across vendors. 31 of the 71 sites that block GPTBot do not block Meta-ExternalAgent.
  • Classic search is untouched. One site blocks Googlebot and none block Bingbot at the root.
robots.txt tokenOperatorSites blocking rootShare of 465
CCBotCommon Crawl8017.2%
BytespiderByteDance7917.0%
GPTBotOpenAI7115.3%
ClaudeBotAnthropic7015.1%
Google-ExtendedGoogle5812.5%
Meta-ExternalAgentMeta5612.0%
PerplexityBotPerplexity5411.6%
Applebot-ExtendedApple4910.5%
AmazonbotAmazon449.5%
ChatGPT-UserOpenAI388.2%
FacebookBotMeta388.2%
Claude-SearchBotAnthropic337.1%
Meta-ExternalFetcherMeta316.7%
OAI-SearchBotOpenAI286.0%
Meta-WebIndexerMeta204.3%
GooglebotGoogle10.2%
BingbotMicrosoft00.0%
Meta-ExternalAdsMeta00.0%
facebookexternalhitMeta00.0%
Sites whose robots.txt disallows the root path for each token, among 465 top-1,000 Tranco domains that allow the wildcard user agent. Source: SaaS Link Building analysis of robots.txt files collected 18 October 2026.

Why the Meta numbers matter now

Meta's crawler documentation describes Meta-ExternalAgent as crawling for foundation model training, Meta-WebIndexer as improving Meta AI search results, and Meta-ExternalAds as crawling to improve advertising and other business products. As Nieman Lab reported, Meta's new terms let it collect content from sites tied to business accounts, and Meta says robots.txt can still be used to manage access. Our data suggests that most large sites have not yet written rules for Meta's ad and search agents, so their default wildcard rules apply.

The split between training and search agents mirrors how the operators describe them. OpenAI's documentation says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Anthropic's help page warns that blocking Claude-SearchBot may reduce a site's visibility in search answers.

Google documents Google-Extended as a control token that does not have its own user agent string.

Methodology

On 18 October 2026 we downloaded the daily Tranco top one million list (list ID 8PX8V) and took the first 1,000 domains. For each domain we requested https://domain/robots.txt, falling back to the www host if the first request failed, with a 12 second timeout and redirects followed. 580 requests returned HTTP 200; 266 returned no response or a connection error (many top domains are CDN, API or tracking hosts with no website), 71 returned 403 and 67 returned 404. We kept the 510 files that returned 200, contained at least one user-agent line and were not HTML pages. Of those, 45 disallow the root for the wildcard user agent; we excluded them, leaving 465 sites that admit crawlers by default.

For each of 19 tokens we parsed the file with an RFC 9309 compliant parser and recorded whether the token could fetch the root path. A site counts as blocking only if the root is disallowed for that token, so partial blocks of sections are not counted. A site can block a token through a named group or through a group that lists several agents. We did not test server or CDN level blocking, so the figures describe stated robots.txt policy, not what crawlers can actually reach.

What to do

  1. Write explicit rules for every AI token you care about. Relying on wildcard rules means new agents such as Meta-WebIndexer or Meta-ExternalAds inherit whatever you set for all crawlers.
  2. Decide on training and search separately. Most large sites that block training crawlers still allow search crawlers. Follow the same logic if AI search citations matter to you.
  3. Review Meta's tokens before 30 October. Set a deliberate policy for Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalAds, and remember that Meta says its fetcher and link preview agents may bypass robots.txt in some cases.
  4. Back robots.txt with server controls for anything sensitive. robots.txt states a preference; it does not enforce one.

Conclusion

Among the most visited sites on the web, about one in seven block the best-known AI training crawlers, far fewer block AI search crawlers, and almost none have written rules for Meta's ad crawler. With Meta's crawling terms changing on 30 October, the gap between how many AI agents exist and how many robots.txt files name them is the clearest action item for technical SEO teams this month.

Questions and answers

How many sites in the sample block GPTBot?

71 of the 465 sites that allow general crawling at the root, or 15.3%, block GPTBot from the root path in robots.txt, based on files collected on 18 October 2026.

How many block Meta's AI crawlers?

56 sites (12.0%) block Meta-ExternalAgent, 20 (4.3%) block Meta-WebIndexer and none block Meta-ExternalAds. 69 sites (14.8%) block at least one of five Meta tokens: Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds and FacebookBot.

Do sites block AI search crawlers as often as training crawlers?

No. In our sample, 43 of 71 GPTBot blockers still allow OAI-SearchBot, and 37 of 70 ClaudeBot blockers still allow Claude-SearchBot.

Which list of domains was used?

The top 1,000 domains of the Tranco research ranking, list ID 8PX8V, downloaded on 18 October 2026.

More releases

  1. Studies

    Do Title Tag Keywords Still Rank? Only 27.5% of 262 Google Titles Held the Exact Query

    Google says titles are a small ranking factor, but our pull of 262 US Google results for 30 queries found only 27.5% of title links held the exact query, and first place rarely had one. DOJ notes and leaked Google documents explain why titles still count.

  2. Updates

    Search Console AI Report Bug Hit Core Update Launch; 3 of 8 2026 Updates Overlapped

    Google has logged an error that cut impressions in Search Console's Generative AI performance report for 9 to 11 November, the window in which the November 2026 core update began. Our timeline of Google's own records shows three of this year's eight ranking updates launched on a day with a logged data error.

  3. Explainers

    Bing's AI-Off Extension Just Adds noAI=1 to Searches; It Has 137 Chrome Users

    Bing documented two ways to hide its AI answers on 11 November: a preview browser extension and a -ai query operator. Our read of the extension's code shows it appends a noAI=1 parameter, ships with AI off, and resets the default search engine and new tab to Bing. It has 137 Chrome users.

Every release

Tell us the site and the goals.

Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.