Meta's New Commercial Terms Let It Crawl Business Websites for AI From 30 October as Publishers Push Back

Ras Al Khaimah, United Arab Emirates, 18 October 2026.

Meta has updated its Commercial Terms so that, from 30 October 2026, it can use automated tools to collect content from websites associated with any business that uses its products, and use that content to develop and train its own products, including AI at Meta. The change was first reported by Nieman Lab on 15 October, and on 16 October the European Publishers Council said it had written to the European Commission asking for urgent scrutiny.

For SEO teams the story is less about one platform's legal language and more about control: who can crawl a site, for what purpose, and whether robots.txt is still the instrument that decides it.

What changed

• Scope widens beyond advertisers. Nieman Lab reports that the crawling permission covers sites "associated with" business accounts. Those accounts are used to run Facebook Pages, Instagram profiles, Messenger and WhatsApp channels, so the clause reaches far more websites than Meta's earlier ad-specific terms did.
• Training is named as a use. According to Nieman Lab, the terms say Meta may use the collected content to train Meta products, and they define AI at Meta as one of those products.
• Test credentials. The terms allow Meta to create test credentials to reach pages. Meta did not tell Nieman Lab whether those credentials could pass paywalls.
• No pre-deadline opt-out. Businesses that disagree are told to stop using Meta's products for business purposes, Nieman Lab reports.
• Consolidation. A Meta spokesperson told Nieman Lab that crawling language had previously been spread across several sets of terms, including those for generative AI in ads, and that the update brings it together without changing how Meta crawls publishers.

Meta's crawlers and how they treat robots.txt

Meta told Nieman Lab that content owners can still manage access with robots.txt. Its crawler documentation lists five user agents, and two of them carry exceptions.

User agent	Stated purpose	robots.txt note
Meta-ExternalAgent	Training foundation AI models and indexing content to improve products	Can be blocked with a Disallow rule
Meta-WebIndexer	Improving Meta AI search results; helps Meta cite and link content	No bypass stated
Meta-ExternalAds	Improving advertising and other business products	No bypass stated
Meta-ExternalFetcher	Fetching individual links a user requests, including for agentic features	May bypass robots.txt
FacebookExternalHit	Link previews for content shared on Facebook, Instagram and Messenger	May bypass robots.txt for security or integrity checks

Meta's documentation also says robots.txt changes can take up to 24 hours to apply, because its crawlers may cache the file.

Why lawyers and publishers are concerned

Legal experts interviewed by Nieman Lab said the contract language could let Meta argue that its commercial terms override blocking instructions in robots.txt. James Grimmelmann of Cornell Law School told the outlet that publishers would have to either trust Meta's narrow reading or stop using business accounts. Matt Topic of Loevy & Loevy said it was unclear whether a court would enforce the provision, but that publishers should assume it would be. Benjamin Sobel of the University of Wisconsin compared the link between business tools and scraping rights to the antitrust idea of tying.

The European Publishers Council is most concerned about wording that extends to content made available through people's use of Meta's services, which it reads as a possible claim on articles that users share. In its statement the council said "A user sharing a news article cannot confer rights they do not hold." It has written to Executive Vice-Presidents Ribera and Virkkunen.

Search Engine Journal reported that Meta spokesperson Andy Stone said on X that the update is about pulling accurate product information for ads and recommendations, not about scraping news sites.

What this means for SEO teams

Nieman Lab notes that virtually every news organisation has at least one Meta business account. Any brand that runs a Facebook Page or Instagram business profile falls under the same terms, so this is a crawl policy question well beyond advertisers.

1. Audit your robots.txt for each Meta token. Decide separately on Meta-ExternalAgent (training), Meta-WebIndexer (Meta AI search and citations) and Meta-ExternalAds. Blocking the indexer may reduce the chance of being cited in Meta AI answers, according to Meta's own description.
2. Check server logs before 30 October. Record current request volumes from each Meta user agent so you can spot any change in behaviour after the terms take effect.
3. Do not rely on robots.txt alone for gated content. Because Meta has not said whether test credentials can pass registration or paywalls, sites with gated material should review authentication and bot management at the server or CDN level.
4. Bring legal in early. Whether to keep business accounts active under these terms is a commercial and legal decision, not just a technical one.

Conclusion

Meta's terms consolidate and widen its crawling rights at the same moment that AI crawlers are becoming a standing item in technical SEO audits. Meta says robots.txt still works and that nothing about its crawling of publishers will change; lawyers and the European Publishers Council are not convinced. With the terms live on 30 October, the practical step for site owners is to set an explicit policy for each Meta user agent now and to watch the logs afterwards.

Questions and answers

When do Meta's updated Commercial Terms take effect?

The updated terms take effect on 30 October 2026, according to reporting by Nieman Lab and a statement from the European Publishers Council.

Which businesses are covered?

Nieman Lab reports that the terms apply to sites associated with Meta business accounts, which are used to manage Facebook Pages, Instagram profiles, Messenger and WhatsApp channels. Advertising is not required.

Can site owners still block Meta's crawlers with robots.txt?

A Meta spokesperson told Nieman Lab that content owners can still manage access with robots.txt. Meta's crawler documentation lists Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalFetcher and FacebookExternalHit, and says the last two may bypass robots.txt in some cases.

Can businesses opt out of the new terms?

According to Nieman Lab, there is no opt-out before 30 October; businesses that disagree are told to stop using Meta's products for business purposes.

###