24.6% of Top Sites' Robots.txt Files Use Fields Google Ignores, 2,000-Domain Crawl Finds

Ras Al Khaimah, United Arab Emirates, 24 October 2026.

Nearly one in four robots.txt files on the world's most visited domains contains fields that Google's crawlers do not read, according to a SaaS Link Building crawl of the Tranco top 2,000 domains on 24 October 2026. Of 1,012 valid files, 249 (24.6%) carried at least one field other than the four that Google's robots.txt specification supports: user-agent, allow, disallow and sitemap.

The crawl follows Google's 23 October update to that specification, which now recommends percent-encoding non-ASCII characters in rule paths so third-party tools can process them. We reported that documentation change this morning.

Key findings

• 249 of 1,012 files (24.6%) include a field Google ignores. The most common is crawl-delay, in 128 files (12.6%).
• 80 files (7.9%) use Host or Clean-param; Yandex documents the second.
• 52 files (5.1%) contain AI or content-use fields. Cloudflare's Content-Signal line appears in 45; 12 use invented names such as LLMS, Agentmap or DisallowAITraining.
• 21 files have rule paths with non-ASCII characters. 13 write at least some of them as raw UTF-8, the form Google now suggests encoding.
• 3 files, on chatgpt.com, launchpad.net and academia.edu, name Perplexity‑User with a non-breaking hyphen (U+2011) instead of an ordinary hyphen, so the line does not match the crawler's published token.
• 4 files still use a robots.txt noindex line, a rule Google stopped handling in 2019.

Which fields Google ignores, and how often they appear

Google's specification says it supports four fields and that others, naming crawl-delay as an example, aren't supported. It also says Google ignores invalid lines and uses only valid ones. None of the fields below breaks a file for Googlebot: Google skips the line and reads the rest. Some are read by other crawlers, so their presence is not necessarily a mistake.

Field	Files	Share of 1,012	Who reads it
crawl-delay	128	12.6%	Not Google; Bing has documented support
host	47	4.6%	Not Google
content-signal	45	4.4%	Not Google; Cloudflare's Content Signals Policy
clean-param	42	4.2%	Not Google; documented by Yandex
Invented AI fields (LLMS, LLM, Agentmap and others)	12	1.2%	No published support found
noindex	4	0.4%	Not Google since 1 September 2019

Crawl-delay is the clearest case of a rule written for one search engine and ignored by another. Bing's webmaster blog described how its crawler applies the directive in a 2009 post; Google's specification lists it as unsupported. The most common values in our sample were 10 (104 lines), 1 (74 lines) and 5 (53 lines). For Googlebot, the place to look is Google's crawl rate documentation, which gained Retry-After header examples on 6 October according to its changelog.

Content-Signal lines are mostly not hand-written. Cloudflare's documentation says it prepends managed directives and its Content Signals Policy to a site's robots.txt when the site turns on that setting and blocks at least one use. Google's parser does not treat them as rules.

The 12 invented AI fields are a different matter. Most are used like a Sitemap line, pointing AI systems at a file: nvidia.com has LLMs.txt, lg.com has LLM and bitdefender.com has Llmstxt, each followed by the address of an llms.txt file, while att.com and hostinger.com use Agentmap to point to an AI catalog in JSON. launchpad.net adds DisallowAITraining: /. We found no crawler documentation that reads any of these fields. Sites that want AI systems to find an llms.txt file are relying on those systems to look for it directly, a file our 19 October study found published on 20.8% of 514 top sites. This matches what Google's Gary Illyes said in a video we covered in our 22 October explainer: robots.txt rules work by naming a crawler's token, not by describing a purpose.

Non-ASCII paths: raw, encoded, or both

Google's updated specification says paths can be written as raw UTF-8 or percent-encoded, that Google converts raw paths into percent-encoded form before matching, and that encoding them yourself makes the file easier for third-party tools to process. Its example treats /foo/bar/ツ and /foo/bar/%E3%83%84 identically.

Only 21 of the 1,012 files contain non-ASCII paths in any form, so this affects a small group: sites whose URLs or blocked search terms use accented or non-Latin characters. Within that group, raw UTF-8 is common:

How non-ASCII paths are written	Files	Rules	Examples
Raw UTF-8 only	9	244 raw	wetter.com, idealista.com, pexels.com
Percent-encoded only	8	75 encoded	yandex.ru, 2gis.com, unsplash.com
Both forms in one file	4	51 raw, 134 encoded	wikipedia.org, wiktionary.org, indeed.com
Total	21	295 raw, 209 encoded

Wikipedia's file shows one way to cover both readers. It lists German maintenance pages such as /wiki/Wikipedia:Löschkandidaten/ in raw form and again as /wiki/Wikipedia:L%C3%B6schkandidaten/. Googlebot treats the two lines as the same rule; a parser that only handles ASCII still gets the encoded copy.

A look-alike hyphen in an AI bot's name

The crawl surfaced one error worth checking for by hand. On chatgpt.com, launchpad.net and academia.edu, a group headed User-agent: Perplexity‑User uses the character U+2011, a non-breaking hyphen, rather than the ASCII hyphen in the token Perplexity publishes. RFC 9309 limits user-agent tokens to letters, hyphens and underscores, defined as ASCII characters, so the line will not match a parser that follows the standard.

We cannot say how the character got there. Our speculation: the table on Perplexity's crawler documentation page labels the agent with the same U+2011 character in its first column, while the code-formatted token beside it uses an ordinary hyphen. Copying the label rather than the code would produce exactly this line. The practical impact is limited: Perplexity's documentation says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules. All three files also name PerplexityBot with a correct token.

Smaller problems

• Byte order mark: 7 files start with a UTF-8 BOM. Google's specification says it ignores the BOM, so this is harmless for Googlebot.
• Lines with no colon: 6 files contain a non-comment line without a field separator, which Google drops as invalid.
• Paths without a leading slash: 4 files have rule paths that start with neither / nor a wildcard. Google's specification says the path value must start with /.
• Relative sitemap URLs: 4 files list a sitemap without a protocol and host; Google asks for a fully qualified URL.
• Size: no file came near Google's 500 KiB limit. The largest was 120,190 bytes.

What this means for SEO teams

• Do not use crawl-delay to slow Googlebot. Keep it for Bing if you need it, but handle Google with 503 or 429 responses and Retry-After during emergencies.
• Know which lines are written for whom. Comment Yandex, Bing and Cloudflare-specific lines so a later editor does not assume they control Google.
• Replace any robots.txt noindex with a robots meta tag or X-Robots-Tag header on a crawlable page, options Google listed when it retired the rule in 2019.
• Drop invented AI fields and name each AI crawler by its exact published token. Our 18 October study lists how top sites block the main ones.
• Check tokens for hidden characters. Search the file for any byte above ASCII outside path values; grep -nP '[^\x00-\x7F]' robots.txt finds them.
• Percent-encode non-ASCII paths, or list both forms as Wikipedia does, if crawlers other than Google matter to you.

Methodology

On 24 October 2026 we took the first 2,000 domains from the Tranco daily list 262N9, generated on 23 October, and requested https://<domain>/robots.txt, falling back to the www host if the first request failed. Requests used a declared research user agent, followed redirects and timed out after 20 seconds. 1,147 domains returned HTTP 200. We excluded 96 responses that were HTML pages and 39 that contained no allow, disallow or sitemap line, leaving 1,012 files. 418 domains returned 403 or 503 to our crawler, and many others are API or infrastructure hosts, so the sample covers sites that served a file to an identified non-search crawler.

Each file was split into lines, comments after # were removed, and the text before the first colon was treated as the field name, compared case-insensitively against Google's four supported fields. Non-ASCII paths were detected in raw form and after percent-decoding. We did not test how each file is applied to Googlebot; Google may receive different responses than our crawler.

Conclusion

Most of the extra lines in top sites' robots.txt files are not errors. They are instructions for Bing, Yandex or Cloudflare's policy layer, sitting in a file that Google reads selectively. The risk is in assuming they apply everywhere: one site in eight sets a crawl delay Google never honours, and a handful still rely on a noindex rule retired seven years ago.

The cases that need fixing are smaller and harder to spot: invented AI fields that no crawler reads, and a look-alike hyphen that turns a valid bot name into one that matches nothing. Google's latest edit to its specification points the same way, towards plain, encoded, standard files that every parser reads the same.

Questions and answers

Which robots.txt fields does Google support?

Google's robots.txt specification lists four fields: user-agent, allow, disallow and sitemap. It says other fields, such as crawl-delay, aren't supported, and that invalid lines are ignored.

How many top sites use crawl-delay in robots.txt?

In our 24 October 2026 crawl, 128 of 1,012 valid robots.txt files from the Tranco top 2,000 domains (12.6%) contained a crawl-delay line. Google does not support the field.

Should I percent-encode non-ASCII paths in robots.txt?

Google says it reads raw UTF-8 paths and converts them to percent-encoded form, but its 23 October 2026 update adds that percent-encoding rule paths makes the file easier for third-party tools to process.

Does Google still support noindex in robots.txt?

No. Google retired code handling unsupported rules such as noindex in robots.txt on 1 September 2019 and pointed site owners to the robots meta tag, the X-Robots-Tag header and 404 or 410 status codes.

Does Perplexity-User follow robots.txt?

Perplexity's crawler documentation says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Its PerplexityBot crawler is controlled separately.

###