NYT Sues Brave Over AI Scraping; 47 of 56 News Sites' AI Bot Blocks Miss Its Crawler

Ras Al Khaimah, United Arab Emirates, 4 November 2026.

The New York Times sued Brave Software in federal court in Manhattan on Monday, 2 November, and the publishers of the New York Daily News, the Chicago Tribune and the Orlando Sentinel filed a separate suit against the browser and search company the same day, TechCrunch reported. Both complaints say Brave built a search index of more than 30 billion pages with a crawler it never named, then sold access to that index to AI companies. For SEO teams, the more practical question is the one buried in the crawler claim: can a site keep Brave out with robots.txt at all? We fetched the robots.txt files of 56 US and UK news sites, the four plaintiffs included, to find out how far their AI bot blocks actually reach.

What the lawsuits allege

According to TechCrunch and TheWrap, the two complaints make overlapping claims:

• The index and the API. Brave's index, which its own marketing puts at more than 30 billion pages, is sold through the Brave Search API. Both suits name five alleged API customers: Anthropic, Perplexity, Mistral, xAI and Cohere. These are the publishers' allegations, and none of the five companies is a defendant.
• An unnamed crawler. The Times describes Brave as a "supplier, dealer, and distributor of stolen copyrighted content" and says publishers cannot block a crawler whose name is never disclosed.
• Ask Brave output. TheWrap reports that the Alden-owned papers say Ask Brave, the company's AI answer feature, produced more than 1,200 words from a Chicago Tribune article on data centre tax breaks and nearly 600 words from a Daily News article on Central Park horse carriages.
• Browser features. Both suits also target a paywall-bypass setting and Brave's practice of blocking publisher ads while selling its own.
• Size of the filings. The Times filed a 72-page complaint and the three papers a 69-page one, according to TheWrap. The Times says cease-and-desist letters it sent to chief executive Brendan Eich in August 2025 and April 2026 went unanswered.

Brave did not respond to requests for comment from either outlet. It has previously defended its indexing as fair use: it sued News Corp in 2025 seeking a court ruling to that effect, and News Corp countersued in July 2026, TheWrap notes. The case numbers, 26-cv-9549 for the Times and 26-cv-9554 for the three papers, were reported by PPC Land.

Why robots.txt can't single out Brave

The crawler claim matches what Brave says about itself. Its Brave Search crawler help page states that the crawler does not advertise a distinct user agent, so that sites which allow only Google are not treated differently. The same page says that if Googlebot is not allowed to crawl a domain or page, Brave's bot will not crawl it either, and that the supported way to keep a page out of Brave's results is a noindex directive, applied once Brave fetches the page again.

Put together, Brave's documented behaviour ties its crawl permissions to Googlebot's. A User-agent: GPTBot or User-agent: ClaudeBot block does nothing to Brave, and there is no Brave token a publisher can disallow on its own. The only robots.txt rule Brave says it honours is a Googlebot disallow, which would also shut out Google.

What we found in 56 news robots.txt files

On 4 November we fetched robots.txt from 56 major US and UK news domains and tested each file against eight named AI agents and Googlebot, using a sample article URL. The headline results:

• 47 of the 56 sites (84%) block at least one of the eight named AI agents from article URLs. All 47 leave Googlebot allowed, so on Brave's own description none of those blocks applies to its crawler.
• 19 sites block all eight agents we tested, and the median site blocks seven. Nine sites block none of them.
• Only 2 sites mention Brave at all: Condé Nast's newyorker.com and wired.com each list a Bravebot user agent in a blocked group. Since Brave says its crawler does not use a distinct user agent, that line has nothing to match (our reading of Brave's help page, not something Brave has stated about these files).
• 2 sites, reuters.com and wsj.com, disallow unknown crawlers through their User-agent: * group. Even that catch-all does not cover a crawler that takes its cue from Googlebot's rules.

The plaintiffs are among the most thorough blockers in the set. Each file runs to dozens of named agents, yet none contains a rule that reaches Brave:

Plaintiff site	Distinct user-agent names in robots.txt	Blocks all 8 AI agents tested	Mentions Brave	Googlebot allowed on articles
nytimes.com	66	Yes	No	Yes
chicagotribune.com	71	Yes	No	Yes
nydailynews.com	71	Yes	No	Yes
orlandosentinel.com	71	Yes	No	Yes

Source: SaaS Link Building analysis of robots.txt files fetched 4 November 2026.

The detail is telling. The Times' file names cohere-ai, two Perplexity agents and five Anthropic agents, and the Tribune-group files even name the Vivaldi browser and xAI's grok-crawler. Brave is the one alleged supplier to those AI companies that the files cannot name, because by its own account there is no Brave name to use.

Context: the stealth crawler debate

Crawler identity has already reached lawmakers: PPC Land reports that more than 300 publishing executives asked Congress to pass a Stealth Bot Prohibition Act. Our earlier counts show how widely named AI bots are now blocked: 128 of 531 top sites block AI bots in robots.txt ahead of a shared AI training opt-out Google says it will honour, and a separate crawl found 12% of 465 top sites block Meta's AI training crawler while none blocked its ads crawler. Each of those controls depends on a crawler announcing itself. Brave's approach does the opposite, which is why the publishers' crawler claim leans on disclosure rather than on robots.txt compliance.

Google is not part of these cases, but its own documentation shows a different model. Google-Extended, which 37 of our 56 sites block, has no HTTP user agent of its own either: Google crawls with its usual agents and treats the robots.txt token as a control. Blocking it keeps content out of Gemini training and grounding, and Google says it does not affect inclusion or ranking in Search. Brave's help page lists no equivalent token. Verifying who is actually crawling matters too: we found 31 of 208 Google crawler IPs had no reverse DNS, a reminder that user-agent strings alone prove little.

What this means for SEO teams

• Don't assume your AI block list covers Brave. If your robots.txt allows Googlebot, Brave's documented behaviour means its crawler can fetch the same pages, whatever else you block.
• A Bravebot line is cosmetic. Keep it if you like, but don't report it to stakeholders as a working control.
• Noindex is the only lever Brave documents, and it is blunt. A noindex meta tag also removes the page from Google. A header or tag aimed only at Brave is not something Brave lists as supported.
• Check server logs by IP and behaviour, not user agent. If you need to know what Brave fetches, a user-agent filter will not find it.
• Watch the case. If the court orders disclosure of Brave's crawler or a way to opt out, the robots.txt picture changes for every site, not just publishers.

Methodology

On 4 November 2026 we requested /robots.txt from 65 large US and UK news domains, trying the www host first and the bare domain second, and kept files that returned HTTP 200 and contained at least one user-agent line. Eight domains returned no usable file. We excluded telegraph.co.uk because the file served to us disallowed Googlebot entirely, which is unlikely to be what Google receives, leaving 56 sites. Each file was parsed with Python's standard robots.txt parser and tested against a sample article path (/2026/11/02/some-article.html) for Googlebot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent, plus an unnamed agent to test each User-agent: * group. "Blocks" means the parser returned disallowed for that path. We also searched each file for the string "brave" and counted distinct user-agent names. Robots.txt files can differ by requester and change at any time; our results describe the files as served to our fetcher on that date. The conclusion that named blocks do not reach Brave rests on Brave's own help page, not on observed crawler traffic.

Conclusion

The Times and the Alden papers are among the most thorough AI bot blockers in our set, and our check shows that their lists stop at Brave. With Brave tying its crawling to Googlebot's permissions and offering no separate user agent, 47 of 56 news sites that block AI bots still leave the door open to the one crawler now accused of feeding those same AI companies.

The lawsuits will decide whether that design is fair use. Until a court or Brave changes it, any site that wants Google traffic has no robots.txt rule that keeps Brave out.

Questions and answers

Who sued Brave and when?

The New York Times sued Brave Software in federal court in Manhattan on 2 November 2026. The publishers of the New York Daily News, Chicago Tribune and Orlando Sentinel filed a separate suit the same day.

Which AI companies do the lawsuits name?

Both complaints allege that Anthropic, Perplexity, Mistral, xAI and Cohere pay for access to Brave's search index through the Brave Search API. These are allegations by the publishers; the five companies are not defendants.

Can I block Brave's crawler in robots.txt?

Not separately. Brave's help page says its crawler does not use a distinct user agent and will not crawl pages Googlebot is not allowed to crawl. Blocking Brave through robots.txt therefore means blocking Googlebot as well.

How do I remove a page from Brave Search?

Brave's help page says to add a noindex directive to the page, which takes effect after Brave fetches the page again. A noindex tag also removes the page from Google.

Does blocking GPTBot or ClaudeBot stop Brave?

No. Those rules apply only to the named agents. In our 4 November check, all 47 of 56 news sites that blocked at least one named AI agent still allowed Googlebot, whose rules Brave says it follows.

###