Google: Emoji User-Agents, Decorative Rules and Custom AI Directives Don't Work in Robots.txt

Ras Al Khaimah, United Arab Emirates, 22 October 2026.

Google Search Central published a video on 21 October 2026 titled "What can you actually put in a robots.txt?", the latest episode of its Non-Canonical Cabinet series, in which Gary Illyes and Cherry Prommawin answer edge-case technical questions from the community. The episode covers ASCII art in robots.txt, how parsers handle formatting mistakes and invented rules, emoji and non-English characters in user-agent tokens and URL paths, and whether robots.txt can stop generative AI from training on a site.

The short answer, which the video description gives as its takeaway, is to keep robots.txt simple. Google's written rules explain why.

What Google covered

The video's chapters run through five questions. Below, each is matched to what Google's robots.txt specification and the RFC 9309 Robots Exclusion Protocol say.

Question in the video	Short answer	Where the rule is written
Can I add ASCII art?	Only as comments; text after # is ignored	Google robots.txt specification
What happens to malformed or custom rules?	Invalid lines are ignored; only valid lines are used	Google robots.txt specification
Can a user-agent token be an emoji or non-English word?	No; tokens are limited to letters, hyphens and underscores	RFC 9309 grammar
Can URL paths contain non-ASCII characters?	Google canonicalises them to percent-encoded form; encoding them yourself is safer	Google specification, RFC 9309 section 2.2.2
Can robots.txt stop AI training?	It controls crawling access by named user agent, not later use of data	Google crawler documentation

Only four fields count

Google's specification says a valid line is a field, a colon and a value, and that Google supports four fields: user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, are not supported. The file must be UTF-8 plain text, Google ignores invalid lines, including a byte order mark at the start of the file, and anything after a # is treated as a comment.

The practical effect is that a mistake is silent. A misspelled field or a rule written in plain language does not raise an error in Googlebot's parser; it simply does nothing, and the content it was meant to protect stays crawlable. Google also caps the file at 500 kibibytes and ignores anything beyond that limit.

Why emoji user-agents fail

RFC 9309 defines a user-agent product token as one or more characters from a narrow set: uppercase and lowercase letters, the hyphen and the underscore. An emoji, an accented letter or a word in a non-Latin script falls outside that grammar, so a group headed by such a token will not match any crawler. Google's specification adds that both the field name and its value are case-insensitive, so Googlebot and googlebot are equivalent.

Non-ASCII paths: Google copes, others may not

Paths are different. Google's specification says paths can be raw UTF-8 or percent-encoded, and that, as defined in RFC 9309 section 2.2.2, Google compares rules against URLs in percent-encoded form and converts raw UTF-8 rule paths automatically. Its example treats Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 identically.

The video's chapter list includes a section on why to percent-encode non-ASCII paths anyway. Search Engine Roundtable's write-up of the episode reports Illyes's reasoning: Googlebot handles UTF-8, but other or older crawlers may not, and a line they cannot parse could be skipped. For sites with German, Japanese or other non-ASCII URLs, writing the encoded form is the version every compliant parser can read.

AI crawlers need their exact token

On generative AI, Google's description frames the question as whether robots.txt can keep AI bots out of private content and stop training. Robots.txt is an access preference: it tells a named crawler which paths it may fetch. A rule that describes content or purpose, rather than naming a crawler's token, will not be understood.

For Google's own AI uses, the relevant token is Google-Extended. Google's crawler documentation says it has no separate HTTP user agent; crawling uses existing Google user agents, and the token is used purely as a control over whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Search. Other AI companies publish their own tokens, and each has to be listed exactly. Robots.txt is also not access control: content that must stay private needs authentication.

We looked at how top sites already use these tokens in our 18 October robots.txt study.

What to do

• Audit for unsupported lines. Remove or comment out anything that is not user-agent, allow, disallow or sitemap if Google is your target, and do not rely on crawl-delay for Googlebot.
• Check every user-agent token against the crawler operator's published documentation, character for character.
• Percent-encode non-ASCII paths so that crawlers other than Googlebot read them the same way.
• Keep decoration in comments and away from lines that carry rules.
• Test changes in Search Console's robots.txt report before and after deployment, and keep the file well under 500 KiB.

Conclusion

The video adds no new rules, but it addresses a pattern Google is clearly seeing: robots.txt files written for people, or for AI bots in natural language, rather than for parsers. Google's specification is unforgiving in a quiet way, keeping valid lines and dropping the rest without warning.

For SEO teams, the safe file is the plain one: four supported fields, exact tokens, encoded paths and comments for anything else.

Questions and answers

Which robots.txt fields does Google support?

Google's robots.txt specification lists four: user-agent, allow, disallow and sitemap. It says other fields, such as crawl-delay, aren't supported.

What happens if a robots.txt line is invalid?

Google's specification says Google ignores invalid lines and uses only valid lines, so an invalid rule has no effect rather than causing an error.

Can I use an emoji as a user-agent token?

No. RFC 9309 limits user-agent product tokens to letters, hyphens and underscores, so an emoji token will not match any crawler.

Does Google read non-ASCII paths in robots.txt?

Yes. Google says it converts raw UTF-8 rule paths to percent-encoded form, so /foo/bar/ツ and /foo/bar/%E3%83%84 are treated identically.

Does blocking Google-Extended remove a site from Google Search?

No. Google's crawler documentation says Google-Extended does not impact a site's inclusion in Google Search; it controls use for Gemini training and grounding.

###