Meta Paused Its Crawling Terms; Only 2 of 485 Top Sites Block Its Ads Crawler
Meta has paused the business terms that would have let it crawl advertisers' websites from 20 November. Our repeat robots.txt crawl of the top 1,000 domains finds little change: 12.8% of 485 sites now block Meta's AI training crawler, up from 12.0% in October, and only 2 block Meta-ExternalAds.

Ras Al Khaimah, United Arab Emirates, .
Meta has paused the updated business terms that would have let it crawl websites tied to its advertisers, but the robots.txt files of the web's largest sites have hardly moved. SaaS Link Building repeated its October robots.txt analysis of the top 1,000 domains on 17 November 2026, three days before the terms were due to start. Of 485 sites that admit crawlers by default, 62 (12.8%) now block Meta-ExternalAgent, Meta's AI training crawler, against 12.0% a month ago. Only 2 block Meta-ExternalAds, the agent Meta says supports its advertising and business products.
The pause was confirmed on 5 November. A Meta spokesperson told Nieman Lab the company is "pausing the changes as we continue to engage with stakeholders", and Meta's Business Help Center now states that the updates to its Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms will not take effect on 20 November 2026. Meta has not given a new date, and its existing terms stay in force.
Key findings
- Little change at the top of the web. Blocking rates for every Meta agent moved by less than 1.5 percentage points in a month, small enough that differences between the two samples could account for them.
- News publishers block Meta's training crawler more than OpenAI's. Among 40 large news and magazine publishers we could assess, 26 block Meta-ExternalAgent and 23 block GPTBot.
- Publishers still let Meta AI cite them. 15 of those 26 publishers block Meta-ExternalAgent but leave Meta-WebIndexer, the crawler behind Meta AI search citations, untouched.
- The ads crawler is almost never named. Only 6 of 485 robots.txt files mention Meta-ExternalAds at all, up from 4 in October, and only one news brand in our sample (hbr.org) blocks it.
- Nobody blocks the link preview fetcher. No site in the sample blocks facebookexternalhit, which builds the title, description and thumbnail when a page is shared on Meta's apps.

Why the terms put robots.txt in the spotlight
Meta first set the updated terms to take effect on 30 October, then moved the date to 20 November after news publishers objected. The contested clause allowed Meta to use automated tools and test credentials to access websites associated with businesses that use its products. The World Association of News Publishers (WAN-IFRA) asked Meta to make robots.txt and other machine-readable reservations of rights binding in the terms rather than optional.
That request matters because robots.txt is the only control Meta documents for its crawlers. Meta's web crawler documentation lists five agents and shows site owners how to disallow them by token. It also says two of them may not follow those rules: facebookexternalhit might bypass robots.txt for security or integrity checks, and Meta-ExternalFetcher, which fetches links at a user's request, may bypass robots.txt rules. Neither the reporting we reviewed nor Meta's crawler page ties the paused terms to a specific user agent.
The Next Web reported that the pause followed a Norwegian court order temporarily blocking the terms and complaints to European and UK regulators.
How large news publishers handle Meta's agents
Because publishers led the protest, we checked 43 well-known news and magazine domains that sit in the Tranco top 1,000. Three dropped out: cnbc.com returned HTTP 403 to our request, and reuters.com and wsj.com disallow the whole site for the wildcard user agent, so every unnamed crawler is already blocked there. The remaining 40 break down as follows.
| Agent | Operator's stated purpose | Publishers blocking it (of 40) |
|---|---|---|
| Meta-ExternalAgent | Training foundation AI models, improving products | 26 |
| GPTBot | OpenAI model training | 23 |
| Meta-ExternalFetcher | User-requested fetches for AI tasks | 17 |
| Meta-WebIndexer | Meta AI search results, citing and linking sites | 12 |
| Meta-ExternalAds | Advertising and business products | 1 |
| facebookexternalhit | Link previews for shared pages | 0 |
Sources: Meta and OpenAI crawler documentation for stated purposes; SaaS Link Building robots.txt collection, 17 November 2026.
Nine of the 26 publishers that block Meta-ExternalAgent do not block GPTBot, while six block GPTBot but not Meta's training crawler. Twelve of the 40 block none of Meta's four AI and ads agents by name. The pattern is selective: most of these publishers refuse model training while leaving open the crawlers that can send citations or link previews back to them.
What this means for SEO teams
For SaaS and B2B sites that advertise on Facebook or Instagram, the pause removes the deadline but not the question. If Meta brings the terms back, the controls available today are the same ones described above.
- Decide per agent, not per company. Blocking Meta-ExternalAgent stops training crawls under Meta's own documentation, while leaving Meta-WebIndexer open keeps the chance of being cited in Meta AI answers. Most publishers in our sample that block Meta's training crawler made exactly that split.
- Write a named group for Meta-ExternalAds if you want a choice. With no named group, the agent follows your wildcard rules. 479 of the 485 top sites we checked have not addressed it by name.
- Do not block facebookexternalhit by accident. It builds link previews when pages are shared on Meta's apps, and Meta says it may bypass robots.txt for integrity checks anyway.
- Check server logs, not just robots.txt. Meta-ExternalFetcher may ignore robots.txt, so server-side bot rules are the remaining option, at the cost of user-requested fetches from Meta AI.
- Watch Meta's Business Help Center. The pause notice is where Meta says it will post a further update. Our earlier coverage of the original crawling terms sets out what the clause allowed.
Methodology
On 17 November 2026, between 17:18 and 17:21 UTC, we requested /robots.txt over HTTPS for each of the 1,000 highest-ranked domains in the same Tranco research list used in our 18 October analysis, retrying on the www host when the bare domain failed. 530 domains returned a plain-text robots.txt file with HTTP 200; responses that were HTML pages, errors or timeouts were excluded. We then excluded 45 sites whose wildcard (User-agent: *) group disallows the root path, leaving 485.
For each agent we applied Google's group-matching rules: a site counts as blocking an agent only if a group names that agent and, under longest-match precedence, the root path "/" is disallowed for it. Agents with no named group were counted as not blocked, since they fall back to the wildcard rules that admit crawlers. Path-level blocks, such as a disallowed /news/ folder, were not counted. The October figures come from our earlier published analysis; the two samples differ in size (465 and 485 sites), which can reflect fetch failures as well as changed files, so small movements should be read as unchanged rather than as a trend. The news subset is a hand-picked list of 43 news and magazine domains present in the top 1,000, not a complete census of publishers.
Conclusion
Meta's pause gives publishers and advertisers time, but our repeat crawl shows the month-long fight over its terms has not changed how the largest sites configure robots.txt. Training crawlers remain the main target, news publishers block Meta's more than OpenAI's, and Meta-ExternalAds, the agent most closely tied to Meta's business products, is still missing from almost every file.
Site owners who want a say over that agent can add a named rule now rather than waiting for Meta's next announcement. We will repeat the crawl if Meta publishes a new effective date.
Questions and answers
Did Meta's new crawling terms take effect on 20 November 2026?
No. Meta's Business Help Center says the updates to its Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms are paused and will not take effect on 20 November 2026. Its existing terms remain in effect.
How many top sites block Meta's AI training crawler?
In our 17 November 2026 crawl, 62 of 485 top-1,000 Tranco sites that admit crawlers by default (12.8%) disallowed Meta-ExternalAgent from the root, compared with 12.0% in our 18 October analysis.
What is Meta-ExternalAds?
Meta's crawler documentation describes Meta-ExternalAds as an agent that supports advertising and other business-related products and services. Only 2 of the 485 top sites we checked block it, and 6 mention it in robots.txt.
Does blocking Meta-ExternalAgent stop Meta AI from citing a site?
Not by itself. Meta documents Meta-WebIndexer as the crawler that improves Meta AI search results, including citing and linking to sites. 15 of 26 large news publishers in our sample that block Meta-ExternalAgent do not block Meta-WebIndexer.
Do all of Meta's crawlers follow robots.txt?
Meta says facebookexternalhit might bypass robots.txt for security or integrity checks, and Meta-ExternalFetcher, which fetches links at a user's request, may bypass robots.txt rules.
More releases
-
Updates
Google Details How to Allow Google-Agent; 24 Top Sites Block a Name Google Never Listed
Google published a guide on 17 November explaining how to let Google-Agent through CDNs, firewalls and bot managers. Our crawl of 1,159 top-site robots.txt files the same day found 24 naming an undocumented GoogleAgent-Mariner token, against 14 naming Google-Agent, which Google says generally ignores robots.txt.
-
Studies
Are AI Overviews Cutting Clicks More Than Google Admits? The Evidence Says Likely
A pre-registered experiment with 1,065 US users found AI Overviews cut organic clicks by about 38% on the searches where they appear, while Google speaks only of stable totals and higher quality clicks. Our pull of 24 mobile results pages found the AI Overview on top of all 22 that showed one.
-
Explainers
Ask Maps Will Phone Local Businesses; 52 of 80 Listings We Pulled Were Phone-Only
Google's Ask Maps can now find beauty, pet care, auto and home repair appointments in the US and call local businesses for the user. In our pull of 80 Google Maps listings across those four categories, 52 offered no online booking link, only a phone number. Google's help pages explain how owners can opt out.
Tell us the site and the goals.
Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.