12% of 465 Top Sites Block Meta's AI Training Crawler in Robots.txt, but None Block Its Ads Crawler

Ras Al Khaimah, United Arab Emirates, 18 October 2026.

Twelve days before Meta's updated Commercial Terms take effect on 30 October, SaaS Link Building analysed the robots.txt files of the 1,000 most popular domains on the Tranco research ranking to see how the web's largest sites treat AI crawlers. Of 465 sites that allow crawlers by default, 71 (15.3%) block OpenAI's GPTBot from the root, 56 (12.0%) block Meta-ExternalAgent and none block Meta-ExternalAds.

The numbers show that AI crawler policy at the top of the web is selective rather than wholesale, and that Meta's newer agents are rarely addressed by name.

Key findings

• Training crawlers are blocked most. CCBot (17.2%), Bytespider (17.0%), GPTBot (15.3%) and ClaudeBot (15.1%) top the list.
• Search crawlers are blocked about half as often. OAI-SearchBot is blocked by 6.0% of sites and Claude-SearchBot by 7.1%. 43 of the 71 GPTBot blockers still allow OAI-SearchBot, and 37 of the 70 ClaudeBot blockers still allow Claude-SearchBot.
• Meta follows the same pattern. 37 of the 56 sites that block Meta-ExternalAgent still allow Meta-WebIndexer, which Meta says helps it cite and link content in Meta AI.
• Meta-ExternalAds is almost invisible. No site blocks it at the root, and only 4 of the 465 robots.txt files mention it at all.
• Blocking is not consistent across vendors. 31 of the 71 sites that block GPTBot do not block Meta-ExternalAgent.
• Classic search is untouched. One site blocks Googlebot and none block Bingbot at the root.

robots.txt token	Operator	Sites blocking root	Share of 465
CCBot	Common Crawl	80	17.2%
Bytespider	ByteDance	79	17.0%
GPTBot	OpenAI	71	15.3%
ClaudeBot	Anthropic	70	15.1%
Google-Extended	Google	58	12.5%
Meta-ExternalAgent	Meta	56	12.0%
PerplexityBot	Perplexity	54	11.6%
Applebot-Extended	Apple	49	10.5%
Amazonbot	Amazon	44	9.5%
ChatGPT-User	OpenAI	38	8.2%
FacebookBot	Meta	38	8.2%
Claude-SearchBot	Anthropic	33	7.1%
Meta-ExternalFetcher	Meta	31	6.7%
OAI-SearchBot	OpenAI	28	6.0%
Meta-WebIndexer	Meta	20	4.3%
Googlebot	Google	1	0.2%
Bingbot	Microsoft	0	0.0%
Meta-ExternalAds	Meta	0	0.0%
facebookexternalhit	Meta	0	0.0%

Why the Meta numbers matter now

Meta's crawler documentation describes Meta-ExternalAgent as crawling for foundation model training, Meta-WebIndexer as improving Meta AI search results, and Meta-ExternalAds as crawling to improve advertising and other business products. As Nieman Lab reported, Meta's new terms let it collect content from sites tied to business accounts, and Meta says robots.txt can still be used to manage access. Our data suggests that most large sites have not yet written rules for Meta's ad and search agents, so their default wildcard rules apply.

The split between training and search agents mirrors how the operators describe them. OpenAI's documentation says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Anthropic's help page warns that blocking Claude-SearchBot may reduce a site's visibility in search answers. Google documents Google-Extended as a control token that does not have its own user agent string.

Methodology

On 18 October 2026 we downloaded the daily Tranco top one million list (list ID 8PX8V) and took the first 1,000 domains. For each domain we requested https://domain/robots.txt, falling back to the www host if the first request failed, with a 12 second timeout and redirects followed. 580 requests returned HTTP 200; 266 returned no response or a connection error (many top domains are CDN, API or tracking hosts with no website), 71 returned 403 and 67 returned 404. We kept the 510 files that returned 200, contained at least one user-agent line and were not HTML pages. Of those, 45 disallow the root for the wildcard user agent; we excluded them, leaving 465 sites that admit crawlers by default.

For each of 19 tokens we parsed the file with an RFC 9309 compliant parser and recorded whether the token could fetch the root path. A site counts as blocking only if the root is disallowed for that token, so partial blocks of sections are not counted. A site can block a token through a named group or through a group that lists several agents. We did not test server or CDN level blocking, so the figures describe stated robots.txt policy, not what crawlers can actually reach.

What to do

1. Write explicit rules for every AI token you care about. Relying on wildcard rules means new agents such as Meta-WebIndexer or Meta-ExternalAds inherit whatever you set for all crawlers.
2. Decide on training and search separately. Most large sites that block training crawlers still allow search crawlers. Follow the same logic if AI search citations matter to you.
3. Review Meta's tokens before 30 October. Set a deliberate policy for Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalAds, and remember that Meta says its fetcher and link preview agents may bypass robots.txt in some cases.
4. Back robots.txt with server controls for anything sensitive. robots.txt states a preference; it does not enforce one.

Conclusion

Among the most visited sites on the web, about one in seven block the best-known AI training crawlers, far fewer block AI search crawlers, and almost none have written rules for Meta's ad crawler. With Meta's crawling terms changing on 30 October, the gap between how many AI agents exist and how many robots.txt files name them is the clearest action item for technical SEO teams this month.

Questions and answers

How many sites in the sample block GPTBot?

71 of the 465 sites that allow general crawling at the root, or 15.3%, block GPTBot from the root path in robots.txt, based on files collected on 18 October 2026.

How many block Meta's AI crawlers?

56 sites (12.0%) block Meta-ExternalAgent, 20 (4.3%) block Meta-WebIndexer and none block Meta-ExternalAds. 69 sites (14.8%) block at least one of five Meta tokens: Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds and FacebookBot.

Do sites block AI search crawlers as often as training crawlers?

No. In our sample, 43 of 71 GPTBot blockers still allow OAI-SearchBot, and 37 of 70 ClaudeBot blockers still allow Claude-SearchBot.

Which list of domains was used?

The top 1,000 domains of the Tranco research ranking, list ID 8PX8V, downloaded on 18 October 2026.

###