20.8% of 514 Top Sites Now Publish llms.txt, a File Google Says Search Can Ignore

Ras Al Khaimah, United Arab Emirates, 19 October 2026.

About one in five of the web's most popular sites now publishes an llms.txt file, according to a SaaS Link Building analysis of the 1,000 highest-ranked domains on the Tranco research ranking, run on 19 October 2026. Of 514 sites that served a valid robots.txt, 107 (20.8%) also served a valid llms.txt. Only 18 (3.5%) used Content Signals, a proposed robots.txt extension for stating AI usage preferences.

The adoption comes despite Google's position. Google's guide to optimising for generative AI in Search says that, for Google Search, sites can ignore tactics such as creating unnecessary AI text files like llms.txt.

Key findings

• One in five top sites has an llms.txt. 107 of 514 sites with a valid robots.txt (20.8%) serve one. Across all 1,000 domains, 108 do.
• Adoption is lowest at the very top. Only 6 of 53 sites ranked 1 to 100 (11.3%) serve the file, against 25.8% of sites ranked 251 to 500.
• Most files follow the proposal's format. 104 of 108 files (96.3%) open with a Markdown H1 heading and 102 contain Markdown links, as the llms.txt proposal describes.
• The full version is rare. 14 domains serve an llms-full.txt file.
• Content Signals remain niche. 18 of 514 sites (3.5%) declare a Content-Signal line in robots.txt. All 18 allow search, and 10 include ai-train=no in at least one group.
• The two travel together. 13 of the 18 Content Signals sites also publish an llms.txt, and 17 robots.txt files point to llms.txt in comments or directives.

Tranco rank	Sites with valid robots.txt	Serve llms.txt	Share	Use Content Signals	Share
1 to 100	53	6	11.3%	1	1.9%
101 to 250	65	12	18.5%	2	3.1%
251 to 500	124	32	25.8%	6	4.8%
501 to 1,000	272	57	21.0%	9	3.3%
Total	514	107	20.8%	18	3.5%

Who publishes the file

In the top 100, the six sites serving llms.txt were cloudflare.com, fastly.net, wordpress.org, digicert.com, opera.com and wordpress.com. Further down, adopters were concentrated among developer platforms, SaaS companies and infrastructure providers, including stripe.com, shopify.com, slack.com, salesforce.com, hubspot.com and nvidia.com. A smaller number of media and consumer brands, among them weather.com, businessinsider.com and foxnews.com, also serve one.

The files vary widely in size. The median was 12,508 bytes, the smallest 415 bytes and the largest 340,724 bytes. A few do not follow the proposal: time.com uses its llms.txt to publish an access policy with a "disallow" rule, closer to robots.txt syntax than to a Markdown summary.

Google and Chrome point in different directions

Google Search and Google's Chrome team treat the file differently. Google Search's guide tells site owners they can ignore llms.txt for Search. Chrome's Lighthouse, by contrast, has an llms.txt audit in its agentic browsing checks. It flags a page when retrieving llms.txt causes a server error and marks the audit not applicable when the file returns a 404, since providing it is optional. The llms.txt proposal, revised to version 2 in August 2026, says AI labs publish the file for their own developer documentation.

Content Signals come from an IETF Internet-Draft by Leah Romm and Michael Tremante. It defines three uses: search (building a search index and showing results, excluding AI summaries), ai-input (feeding content into AI models for grounding or generative answers) and ai-train (training or fine-tuning models). Among the 18 adopters, 15 set ai-input=yes in at least one group and 4 set ai-input=no.

The data lands days after Cloudflare said it will score Verified bots on whether they honour robots.txt, which makes the rules a site publishes there more consequential than an llms.txt file that Google Search says it does not need.

Methodology

On 19 October 2026 at about 03:20 UTC we downloaded the daily Tranco top one million list (list ID 467YX) and took the first 1,000 domains. For each domain we requested https://domain/robots.txt, /llms.txt and /llms-full.txt, retrying on the www host when the first request did not return HTTP 200, with a 12 second timeout and redirects followed. 705 domains returned an HTTP response to at least one request; many top domains are CDN, API or tracking hosts with no public website.

A robots.txt counted as valid if it returned 200, was not an HTML page and contained at least one user-agent line (514 domains). An llms.txt counted as valid if it returned 200, was not HTML, JSON or an image and held more than 20 characters of text. Of 248 llms.txt requests that returned 200, 140 were excluded: 137 were HTML pages, typically homepages or soft 404s, and three were an error message, an image or a near-empty file. 14 valid files were reached through a redirect to another hostname owned by the same company. We counted a site as using Content Signals if its robots.txt contained a line starting with "Content-Signal:". We did not test whether any AI system reads these files.

What to do

1. Do not expect llms.txt to move Google rankings. Google Search says it is not needed. Treat it as documentation for AI agents and developer tools, the use the llms.txt proposal describes.
2. If you publish one, keep it valid. Serve it as plain text or Markdown with a 200 status, avoid returning an HTML page at /llms.txt, and avoid server errors, which Lighthouse flags.
3. Put AI preferences where crawlers look. robots.txt remains the file crawlers and Cloudflare check. If you want to separate search from training, name the agents and consider Content Signals.
4. Check for soft 404s. 137 top domains answered /llms.txt with a 200 status and an HTML page, which tools can misread as a real file.

Conclusion

llms.txt has spread to a fifth of the most visited sites, led by developer and SaaS platforms, even though Google Search says it can be ignored and Chrome's Lighthouse treats it as optional. Content Signals, which put AI usage preferences into robots.txt, are used by fewer than one in 25 of the same sites. For SEO teams, robots.txt remains the file with measurable consequences.

Questions and answers

How many top sites have an llms.txt file?

107 of the 514 sites in our sample with a valid robots.txt file (20.8%) served a valid llms.txt file on 19 October 2026. Across all 1,000 domains checked, 108 did.

Does Google use llms.txt for Search?

Google's guide to optimising for generative AI in Search says sites can ignore tactics such as creating unnecessary AI text files like llms.txt for Google Search. Separately, Chrome's Lighthouse includes an llms.txt audit in its agentic browsing checks and marks it not applicable when the file is absent.

What are Content Signals?

Content Signals are a proposed robots.txt extension, set out in an IETF Internet-Draft, that lets sites state preferences for three uses: search, ai-input (content used to ground AI answers) and ai-train (model training).

How common are Content Signals among top sites?

18 of 514 sites (3.5%) in our sample had a Content-Signal line in robots.txt. All 18 allowed search, and 10 included ai-train=no in at least one group.

###