Google Adds Retry-After Guidance to Its Crawl Rate Documentation

Ras Al Khaimah, United Arab Emirates, 13 October 2026.

Google updated its Reduce the Google crawl rate documentation on October 6, 2026. According to Google's crawling documentation changelog, the emergency crawl rate reduction section was restructured and now includes information and examples for the Retry-After HTTP header.

What changed

The guide already told site owners that the fastest way to cut Google's crawling for a short period, such as a couple of hours or one to two days, is to return a 500, 503 or 429 status code instead of 200. The updated version adds that when returning a 503 or 429, a site can also send a Retry-After header, as defined in RFC 9110, to tell Google's crawlers when they may retry. The header accepts either a delay in seconds or an absolute date and time in UTC. Google's examples:

HTTP/1.1 503 Service Unavailable
Retry-After: 120

HTTP/1.1 503 Service Unavailable
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Google stresses that support itself is not new. The header was already covered in its guide to temporarily pausing or disabling a website. The change brings it into the crawl rate guide so it is easier to find when a site needs to reduce crawler traffic urgently. Search Engine Roundtable, which first reported the update, noted that most of the other edits reorganize existing content.

How the error-code method behaves

Per Google's documentation, when its crawlers see a significant number of 500, 503 or 429 responses:

• Crawl rate drops across the whole hostname, including URLs that still return normal content, not only the URLs returning errors.
• Crawl rate recovers automatically once the errors subside.
• The method is meant for short periods. Google warns against using it for longer than one to two days, because URLs that return these codes for multiple days may be dropped from the index.

The guide also flags the side effects of slowing Google down at all: fewer new pages discovered, existing pages refreshed less often, removed pages lingering in the index, and, for Google Ads, campaigns that may be paused or ads that may not serve. If serving errors is not practical, site owners can file a special request to report an unusually high crawl rate. Increases cannot be requested, and requests can take several days.

Why it matters now

The timing data Google's Gary Illyes showed at Search Central Live Deep Dive Europe on October 2 explains why a precise signal helps. According to the attendee recap, crawl capacity can drop within seconds when Google backs off a struggling server, but updating it typically takes from four hours to two weeks, and one to three weeks in recovery cases. A Retry-After value gives the crawler an explicit time to return rather than leaving it to infer one from repeated errors.

Crawl load is also a growing operational concern. Google's July 2026 update to its crawl budget guide stated that all of its crawlers share a single crawl capacity limit per site, and speakers in Barcelona described CDNs blocking more bot traffic, sometimes including Googlebot by mistake.

What to do

• Use 503 or 429 with Retry-After for short, planned or emergency load reduction, such as maintenance windows or traffic spikes.
• Keep the window short. Hours or a day or two is the intended range; longer periods risk URLs being dropped.
• Check your CDN and web application firewall rules so that rate limiting returns 429 with a sensible Retry-After, rather than a challenge page served with a 200 status, which Google can read as a thin soft 404.
• Monitor the Crawl Stats report in Search Console afterward to confirm the crawl rate returns to normal.

Conclusion

This is a documentation change, not a change in how Googlebot behaves. But it removes ambiguity at the moment it matters most: when a server is under strain and someone needs a reliable way to slow Google down without damaging indexing. Pairing a temporary 503 or 429 with Retry-After is now the clearly documented way to do it.

Questions and answers

Does Google support the Retry-After header?

Yes. Google's documentation says that when a site returns a 503 or 429 status code, it can include a Retry-After header with a delay in seconds or a UTC date and time to indicate when Google's crawlers can retry.

When did Google update the crawl rate documentation?

October 6, 2026, according to Google's crawling documentation changelog.

Will returning 503 errors hurt my rankings?

Not for short periods. Google recommends the method for a few hours or one to two days. Serving errors for longer can lead to URLs being dropped from the index.

###