Google Warns CDN Moves Can Swamp Origins; 154 of 346 Top Homepages Skipped the Cache

Ras Al Khaimah, United Arab Emirates, 12 November 2026.

Google has warned site owners that putting a site behind a content delivery network (CDN) can briefly increase the load on the very server the CDN was meant to protect. In a Search Central blog post published on 11 November, tied to the fifth episode of its "The Non-Canonical Cabinet" video series, Google explains that faster edge responses encourage its crawlers to request more pages, while a new CDN's empty cache sends many of those requests straight back to the origin.

To see how much of that shield is in place on large sites today, we requested the homepages of the top 1,500 domains in a public popularity ranking twice in a row and read the cache headers they returned. Of 346 sites that exposed a cache-status header, 154 (45%) did not serve their homepage from the edge cache on either request.

What Google said

The post walks through why a CDN switch can look like a crawl surge, and what to do about it:

• Speed raises crawl rate. Google's crawlers watch how fast and how reliably a host responds. When DNS starts pointing to a fast edge network, response times fall and Google's systems read that as a healthier server, so the crawl capacity limit rises, provided there is crawl demand.
• A cold cache passes requests through. The first request for any URL at an edge location is a cache miss, so the edge fetches it from the origin. Human visitors cluster on a few popular pages that warm up quickly; crawlers work through the whole URL inventory, so they hit far more uncached URLs.
• Blanket bot protection backfires. If a CDN or web application firewall challenges all automated traffic, Google's crawlers cannot pass a CAPTCHA. Google says pages left behind such walls will start to drop out of Search as its crawlers back off.
• Three fixes. Allow verified bots in CDN and firewall settings, using Google's published IP ranges to verify its crawlers; pre-warm caches for high-traffic and frequently crawled URLs before cutting over; and, if the origin is still struggling, return 429 or 503 for a short window, which Google puts at a few hours or one to two days at most, as its guide to reducing crawl rate describes.

None of this is a policy change. Google's crawl budget and crawl rate documentation already covered the mechanics, and the Retry-After detail was added to the crawl rate guide in October, which we covered when Google added Retry-After guidance. What is new is Google stating plainly that a CDN migration is a known trigger for an origin crawl spike, and that blanket bot rules switched on in response can knock pages out of Search.

What our cache check found

Google's warning rests on one assumption: that the CDN will eventually serve most crawler requests from cache. For HTML pages, that is often not how CDNs are set up. Of the 1,500 domains we tested, 666 returned a normal homepage (HTTP 200, no challenge page) to a desktop browser user agent. Response headers tied 427 of those 666 (64%) to an identifiable edge network: Cloudflare (162), Amazon CloudFront (90), Akamai (51), Fastly (51), Google (43) and smaller providers.

346 of the 666 returned a cache-status header we could read. Across two requests sent seconds apart:

Outcome over two back-to-back requests	Sites	Share
Served from cache both times	180	52%
Miss first, then hit (cache warmed by our first request)	10	3%
Not served from cache either time (miss, dynamic, bypass or no-cache status)	154	45%
Other combinations	2	1%

The split differs by provider. Cloudflare sites were the most likely to send the homepage to origin, 80 of 152. That matches Cloudflare's own documentation, which says its CDN does not cache HTML or JSON by default; Cloudflare reports such responses with a DYNAMIC cache status. On CloudFront, 46 of 90 homepages were not served from cache either time. Among the larger groups, Fastly sites cached most often, with 36 of 50 hitting the cache on both requests.

The point for SEO teams: when HTML is not cached at the edge, every Googlebot fetch of a page still reaches the origin, cold cache or warm. A CDN makes responses faster, which Google says lifts crawl rate, but it absorbs none of the HTML load unless caching rules for pages are switched on. A homepage is also the most likely page on a site to be cached; deep URLs that only crawlers visit are less likely to be warm.

Spoofed Googlebot requests were often refused

We also requested each homepage with Googlebot's user agent string. These requests came from our own servers, not from Google's IP ranges, so they test how sites treat an unverified claim to be Googlebot, not how they treat Google. Of the 666 sites that served the browser request normally, 58 (9%) refused the fake Googlebot or served it a challenge: 41 with a 403, six with a challenge page, and the remaining 11 with server errors, a 429 or no response. Akamai-fronted sites refused it most often, 16 of 51.

That is the behaviour Google recommends: trust the IP, not the user agent string. It is also why spoofed Googlebot tests from your own laptop prove little. When we checked last month, 31 of 208 Google IPs had no reverse DNS, so allow-lists built on Google's published IP ranges are safer than ones built only on reverse lookups.

What to do before and during a CDN move

• Decide whether HTML will be cached. Check your CDN's default rules. If HTML passes through, expect the origin to carry every crawler fetch of every page, and size it for a higher crawl rate after cutover.
• Pre-warm from your logs. Google suggests fetching high-traffic pages, upcoming seasonal pages and the most frequently crawled URLs through the CDN at a controlled pace before switching DNS. Server logs filtered to verified Googlebot requests give you the most-crawled list.
• Turn on verified bot allow-listing before any bot rule. Most CDNs have a managed setting for known or verified bots. Test with the URL Inspection tool in Search Console rather than with a spoofed user agent.
• Exempt robots.txt from challenges. In our check, 60 of the 666 sites returned a 403 or challenge for robots.txt to the spoofed Googlebot request. If a rule catches the real crawler on robots.txt, the impact reaches every URL on the host.
• Use 429 or 503 as a brake, briefly. Google's guidance talks of a couple of hours or one to two days. Its crawl rate guide warns that if Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index.
• Watch requests and response times together. Google's crawl budget guide says stable or improving response times raise the crawl capacity limit. After cutover, expect falling response times to be followed by a rising Googlebot request count; server logs show it first.

For a closer look at how CDNs are starting to grade bots on robots.txt compliance, see our report on Cloudflare's verified bot trust bands.

Methodology

On 12 November 2026 we took the top 1,500 domains from that day's Tranco list, a public ranking of popular domains. For each we sent two GET requests to the HTTPS homepage with a desktop Chrome user agent, a third with Googlebot's user agent string, and a fourth for robots.txt with Googlebot's user agent string, following redirects, from one cloud server. Many entries in the list are not websites (API, ad and DNS domains), and others blocked our server; 666 returned HTTP 200 without a challenge page to the browser request and form the base for this analysis. We identified edge networks from response headers (for example cf-ray, x-amz-cf-id, x-served-by and Akamai or Google server headers) and read cache status from cf-cache-status, x-cache, x-vercel-cache, cdn-cache and similar headers. A hit on both requests counts as cached; any combination of miss, dynamic, bypass, expired or no-cache values on both counts as not served from cache. Results reflect one location and one moment; edge caches differ by region, and sites can cache differently for logged-in users or other pages.

Conclusion

Google's message is that a CDN migration can raise crawl rate before it lowers origin load, and that blanket bot blocking during that window can cost visibility in Search. Our check adds a second point: on close to half of the large sites that expose cache headers, the homepage HTML is not being cached at all, so the origin carries crawler traffic indefinitely rather than just during warm-up. Teams planning a move should settle their HTML caching rules, verified bot settings and a short 503 or 429 fallback before they switch DNS, not after the first spike in their logs.

Questions and answers

Why does crawling increase after moving to a CDN?

Google says its crawlers raise the crawl capacity limit when a host responds quickly and consistently. A CDN edge usually responds much faster than the old server, so Google's systems treat it as a healthier host and gradually request more pages, provided there is crawl demand.

Why does a new CDN not protect the origin server straight away?

A new CDN starts with an empty cache. The first request for each URL at each edge location is a cache miss that the edge fetches from the origin. Crawlers request many URLs that human visitors rarely open, so they cause more misses than normal traffic.

Should I turn on bot protection if crawler traffic spikes?

Not as a blanket rule. Google says its crawlers cannot pass CAPTCHA challenges and that pages behind such walls start to drop out of Search. It recommends allowing verified bots in CDN and firewall settings and targeting specific unwanted bots instead.

How can I slow Googlebot down during a CDN migration?

Google's crawl rate guide says to return 500, 503 or 429 status codes for a short period, such as a couple of hours or one to two days. If Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index.

Does Cloudflare cache HTML pages by default?

No. Cloudflare's documentation says its CDN does not cache HTML or JSON by default. In our check, 80 of 152 Cloudflare-fronted sites did not serve their homepage from cache on either of two requests.

###