Google Briefly Shields Key URLs From 4xx Drops; VLC Still Sends 418 to 4 AI Bots

Ras Al Khaimah, United Arab Emirates, 5 November 2026.

Google Search can keep an important URL in its index for a short time after it starts returning a 4xx error, Google's Gary Illyes said this week, explaining why VideoLAN's VLC site stayed in results without a snippet or favicon. In a reply under VideoLAN's 3 November LinkedIn post, in which the project asked for SEO help and said it had not changed anything, Illyes said the site had been serving HTTP 418 ("I'm a teapot"), which Google handles like most other 4xx codes, as a page that was not found. Because of the page's importance, he wrote, Search "temporarily protected the URL from falling out of the index right away." The exchange was first reported by Search Engine Roundtable on 5 November.

That grace period is not in Google's documentation. Google's HTTP status code guide says URLs that are already indexed and start returning a 4xx code are removed from the index. We checked what VideoLAN's server returns now, how its listings look in Google, and how 792 of the web's most visited homepages answer crawler user agents they want to turn away.

What changed

• An undocumented exception. Illyes described a temporary hold on removal for URLs Google considers important. He did not say how long the hold lasts or how importance is judged.
• The cost of the hold. While a URL is protected, Google has no current content for it, so the result shows only the link and the last good title it had, with no snippet and no favicon, according to Illyes.
• The recovery. Illyes said the site was returning 200 OK with content again and the listing should correct itself.
• 418 is not special. Google's guide says all 4xx codes except 429 are treated the same, and it notes that exotic codes such as 418 are not covered.

Our check of videolan.org

On 5 November at 21:17 UTC we requested the VideoLAN homepage with 17 crawler user agents from a non-Google IP address. Googlebot (desktop and smartphone), Googlebot-Image, GoogleOther, Google-InspectionTool, Bingbot, Applebot, ClaudeBot, CCBot, Amazonbot, Bytespider, DuckDuckBot and Meta's crawler all received 200. Four received 418 with a 14-byte "I am a teapot" body: GPTBot, OAI-SearchBot, ChatGPT-User and PerplexityBot. For GPTBot, the same 418 came back for the /vlc/ page and the favicon file.

So the teapot response is still live on the server, aimed at some AI crawler user agents. We cannot see what Googlebot received during the incident, and neither VideoLAN nor Google has said which rule caught Google's crawler. Our reading, which is speculation, is that a filter meant for AI bots matched Google's requests for a period.

We then pulled five Google results pages for VLC searches through a SERP API on 5 November ("vlc" in the US and UK, "vlc media player", "videolan" and "vlc download" in the US), plus a site: query for www.videolan.org. Results gathered this way can differ from what users see.

What we counted	Result
Organic results across the five searches	44
Results on videolan.org hosts	19
videolan.org results shown with a favicon	1 of 19 (www-test.videolan.org)
Results on other hosts shown with a favicon	25 of 25
Appearances of the bare homepage or /vlc/	0
Top 10 for site:www.videolan.org	9 language copies of the homepage (index.uz.html, index.fa.html and others) and 1 project page

Source: SaaS Link Building SERP pulls, 5 November 2026.

Language copies and subdomains such as download.videolan.org and code.videolan.org filled the space where the main pages would normally sit. That fits a site whose key URLs are still being re-crawled after the 418 period, though Google has not commented on the current rankings and other factors, including the site's language negotiation, could also play a part.

How top sites turn crawlers away

VLC's teapot is rare. We requested the homepages of the 1,500 highest-ranked domains in the Tranco list with a Chrome user agent and three crawler user agents. 792 returned 200 to Chrome, and we compared how those 792 answered the bots.

• GPTBot: 101 of 792 homepages (12.8%) returned a 4xx code. 86 used 403. The rest were 402 (4), 406 (4), 429 (4), and one each of 401, 451 and a non-standard 471.
• PerplexityBot: 70 homepages returned a 4xx code, 56 of them 403.
• Googlebot user agent from our non-Google IP: 60 returned a 4xx code, 55 of them 403. These sites appear to check whether a request claiming to be Googlebot really comes from Google, as Google recommends. Real Googlebot would likely get through.
• Selective blocking: 65 homepages refused GPTBot while serving 200 to the Googlebot user agent, the same split VideoLAN set up.

What each refusal code means to Google

Code	Seen in our crawl	Google's documented handling
403, 401, 402, 404, 406, 451, 471, 418	All except 418	Treated as not found; indexed URLs are removed (Illyes now says important ones may get a short hold)
429	Yes	Treated as a server error: crawling slows, indexed URLs are kept for a while
503	Yes (4 homepages, Googlebot user agent)	Crawling slows; indexed URLs are preserved, then dropped if errors persist

Sources: Google crawling documentation; SaaS Link Building crawl, 5 November 2026.

The 418 code has an odd history. It began as an April Fool's joke in RFC 2324, and the current HTTP standard, RFC 9110, lists it as unused and reserved because it was deployed as a joke so often. To Google it is just another not-found response.

What the LinkedIn thread got wrong

Several commenters under VideoLAN's post blamed the missing favicon on its size, saying 16x16 was below a 48x48 minimum. Google's favicon guidelines set the minimum at 8x8 and only recommend going above 48x48. The same page says Googlebot must be able to crawl the home page and Googlebot-Image must be able to fetch the favicon file. When a server refuses those requests, no icon size will fix it, which matches Illyes's explanation.

What SEO teams should do

1. Test bot rules against every Google user agent. If you filter AI crawlers by user agent, request your key pages as Googlebot, Googlebot-Image and Google-InspectionTool, and use URL Inspection in Search Console to see what Google actually received. A loose match on "bot" or a shared rule can catch Google.
2. Pick one refusal code and keep it standard. 403 is what most large sites use. Google treats any 4xx except 429 as content that does not exist, so a block that hits Googlebot removes pages, whatever the code.
3. Use 503 for downtime, not 4xx. If you need Google to back off temporarily, 503 or 429 keeps indexed URLs for a while; we covered this in Google's advice on serving 503 during PHP patching.
4. Do not rely on the grace period. Illyes described it for a site he called one of the most important on the internet. Most sites should expect the documented behaviour.
5. Watch for snippet-less listings. A result with a title but no snippet or favicon is a sign Google can no longer fetch the page. Check server logs for 4xx responses to verified Google IPs, as in our Googlebot verification test.

Related: our study of how top sites handle dead pages found 15% of 546 top sites did not return a 404 for missing URLs, and Google's favicon rules for redirects explain another common cause of missing icons.

Methodology

On 5 November 2026 between 21:17 and 21:27 UTC we requested videolan.org with 17 crawler user-agent strings using curl from a cloud server (not a Google IP). For the wider crawl we took the top 1,500 domains from the Tranco list, requested https://domain/ with up to 5 redirects and a 15-second timeout using four user agents (Chrome 130, Googlebot smartphone, GPTBot 1.2, PerplexityBot 1.0), and kept the 792 domains whose homepage returned 200 to Chrome. Status codes are the final response after redirects. Because requests did not come from the crawlers' own IP ranges, the results show how sites treat these user-agent strings, not what the real crawlers receive. SERP data came from six Google searches (five topical searches and one site: query) run through a SERP API on 5 November 2026; favicon presence is the favicon field in that data.

Conclusion

Illyes's comment adds a rare exception to Google's documented 4xx handling: for a small set of important URLs, removal from the index can wait while Google re-checks. The VLC case also shows the cost, with listings stripped to a bare link and, in our pulls, the main pages still absent while language copies stand in.

The practical lesson is older than the teapot. Bot filters that rely on user-agent strings are easy to get wrong, and when they catch Googlebot, Google reads the refusal as a sign the page is gone.

Questions and answers

Does Google remove pages that return a 4xx status code?

Google's crawling documentation says indexed URLs that start returning a 4xx code (other than 429) are removed from the index. Gary Illyes said on LinkedIn that Search can temporarily protect important URLs from being dropped right away, as it did for VideoLAN's VLC site.

How does Google treat HTTP 418 I'm a teapot?

Google's status code guide does not cover 418, but it says all 4xx codes except 429 are treated the same. Illyes said Google handles 418 like most other 4xx errors, as not found.

Why did the VLC result lose its snippet and favicon?

According to Illyes, while Google held the URL in the index it no longer knew what was on the page, so it could show only the link and the last good title, without a snippet or favicon.

What minimum favicon size does Google require?

Google's favicon guidelines require a square image of at least 8x8 pixels and recommend going larger than 48x48 pixels. Googlebot must also be able to crawl the home page and Googlebot-Image must be able to fetch the favicon file.

Which status code should a site use to pause Google crawling?

Google's documentation says 503 and 429 make its crawlers slow down while indexed URLs are preserved for a time, whereas other 4xx codes are treated as not found.

###