Gary Illyes Says AI Made Fake Googlebots Worse; 31 of 208 Google IPs Have No Reverse DNS

Ras Al Khaimah, United Arab Emirates, 30 October 2026.

Google's Gary Illyes has said AI is the main reason fake Googlebot traffic has become a bigger problem, in a podcast interview reported by Search Engine Roundtable on 30 October 2026. Asked why crawler impersonation has grown, Illyes answered "I think it is AI", tying it to the scraping demand created by model training. He repeated Google's long-standing advice: do not trust a user-agent string, and verify a crawler through reverse DNS and IP checks.

To see how well that advice works in practice, SaaS Link Building ran Google's own verification method against 208 addresses drawn from the five IP range files Google publishes for its crawlers and fetchers. Googlebot's ranges passed cleanly. One file did not: 31 of the 40 sampled addresses in Google's Google Cloud user-triggered fetcher ranges had no reverse DNS record at all, so the manual check Google describes could not confirm them.

What Illyes said

According to the Search Engine Roundtable summary of the interview, Illyes made these points:

• Demand for AI training data has pushed up scraping, and many scrapers claim to be Googlebot because that name can get them past blocks.
• Firewalls and CDNs catch many fake crawlers, but not all of them.
• Site owners should verify crawlers with reverse DNS and IP checks rather than the user-agent alone.
• Robots.txt is advisory: legitimate crawlers follow it, bad actors can ignore it.
• Google offers the Google-Extended token so sites can keep Search crawling while limiting use of their content for Gemini training and grounding.
• Web Bot Auth, which signs requests cryptographically, could make verification more reliable, and he is exploring an early idea for a crawler reputation system.
• Most small sites do not need to act right away; they should first measure whether bot traffic is a real problem.

None of this changes Google's documentation, which already warns on its list of common crawlers that the HTTP user-agent string can be spoofed. What is new is a Googler on record naming AI as the cause. It fits a pattern we have tracked this month, from Cloudflare's plan to score verified bots on robots.txt compliance to the share of top sites that block AI training crawlers in robots.txt.

Google's two verification methods

Google's page on verifying requests from Google crawlers and fetchers describes two routes. The manual route is a reverse DNS lookup on the requesting IP, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, and then a forward DNS lookup to confirm the hostname resolves back to the same IP. Google says this is enough for most use cases. The automatic route matches the IP against five JSON files of IP ranges.

The same page lists a hostname pattern for each group of crawlers, shown below.

Group	IP range file	Documented reverse DNS pattern	Follows robots.txt?
Common crawlers (Googlebot)	common-crawlers.json	crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com	Always, for automatic crawls
Special-case crawlers (for example AdsBot)	special-crawlers.json	rate-limited-proxy-***-***-***-***.google.com	May or may not
User-triggered fetchers	user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json	***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com	No, because a user requested the fetch

What our DNS test found

We picked random prefixes from each file, chose a random address inside each, and ran the reverse and forward lookups Google describes. The results split sharply by file.

The Googlebot file behaved as expected. All 40 sampled addresses had a reverse DNS record that resolved back to the same IP. Thirty-seven matched the documented googlebot.com patterns. The other three, all in 34.x ranges, resolved to bc.googleusercontent.com hostnames. They still pass Google's domain test, because googleusercontent.com is one of the three accepted domains, but a filter written to accept only googlebot.com would reject them.

The special crawlers file, the google.com user-triggered fetcher file and the user-triggered agents file all verified fully, with every hostname matching the documented rate-limited-proxy or google-proxy pattern. One special crawler lookup failed on our first pass and succeeded on a retry, so we count it as verified.

The Google Cloud user-triggered fetcher file is where DNS verification breaks down. Of 40 sampled addresses, 31 returned no PTR record. Seven resolved to gae.googleusercontent.com hostnames and confirmed in both directions, but in a dotted, reversed format (for example, 54.198.178.107.gae.googleusercontent.com) rather than the dash format Google's page shows. The remaining two pointed to bc.googleusercontent.com names that did not resolve forward.

The IP files themselves

All five files were regenerated on 29 October 2026, according to their creationTime fields. Their totals are unchanged from our 17 October analysis of Google's crawler IP ranges: 2,181 prefixes, 1,104 of them IPv4. Googlebot's file still lists 317 prefixes, 122 of its 170 IPv4 blocks inside 66.249.0.0/16.

The stale legacy file we flagged then is still there. Four of the five old addresses under /search/apis/ipranges/ redirect to the new /crawling/ipranges/ path, but the old user-triggered-agents.json still returns HTTP 200 with a copy generated on 3 March 2026. Google's 31 March announcement of the move said the old locations would be redirected within six months.

What this means for SEO teams

• Use the IP files for anything automated. Reverse DNS works for Googlebot, but it cannot confirm most Google Cloud fetcher addresses. Matching against the five JSON files at /crawling/ipranges/ covers every group.
• Accept all three Google domains. A rule that only trusts googlebot.com would have rejected 3 of our 40 genuine Googlebot-range addresses. Google's documented test is googlebot.com, google.com or googleusercontent.com, plus a forward lookup.
• Do not match hostnames too strictly. Patterns copied from Google's page would miss the dotted gae.googleusercontent.com format we saw on every confirmed Google Cloud fetcher address.
• Treat user-triggered fetchers differently. Google says they ignore robots.txt. Allowlisting all five files lets them through, so decide whether that is what you want before blocking or allowing.
• Check logs before buying tools. Following Illyes's advice, measure how much traffic claiming to be Googlebot fails verification before adding firewall rules.
• Use Google-Extended for AI training choices. Blocking a spoofed Googlebot is a security task; limiting Gemini training is a robots.txt choice made with the Google-Extended token, which Google says does not affect inclusion in Search.

Web Bot Auth is still experimental

Google's Web Bot Auth documentation, added in May, says Google is testing the IETF draft with some AI agents hosted on its infrastructure. It states that not every Google user agent uses the protocol and that Google does not yet sign every request from those that do, and it tells site owners to keep relying on IP addresses, reverse DNS and user-agent strings. For now, signed requests are a supplement, not a replacement.

Methodology

On 30 October 2026, between about 14:10 and 14:30 UTC, we downloaded the five JSON files linked from Google's verification page from developers.google.com/static/crawling/ipranges/ and requested the matching legacy file names under /static/search/apis/ipranges/, recording status codes and redirects. We counted prefixes with Python's ipaddress module. For the DNS test we drew 40 random IPv4 prefixes from each of the four larger files and, because the user-triggered agents file has only 12 IPv4 prefixes, sampled each of those four times (48 lookups). We chose one random address inside each prefix, queried its PTR record through Google Public DNS over HTTPS, then queried the A record of the returned hostname and checked that it contained the original address. Failed lookups were retried up to three times. We compared each hostname with the patterns on Google's verification page. These are addresses inside published ranges, not addresses we saw crawling a site, so an address with no PTR record may simply not be in use; the result shows that the DNS method cannot confirm those addresses, not that Google traffic from them is fake.

Conclusion

Illyes's comments put a Google name to something many site owners see in their logs: more bots claiming to be Googlebot, driven by AI scraping. Google's answer is unchanged, and for Googlebot itself the reverse DNS check works well.

The weak point is outside Googlebot. Most sampled addresses in Google's Google Cloud fetcher ranges could not be confirmed by DNS, and the hostnames that did resolve used a format Google's page does not show. Teams that verify Google traffic at scale should rely on the published IP files and refresh them often.

Questions and answers

What did Gary Illyes say about Googlebot impersonation?

In a podcast interview reported on 30 October 2026, Illyes said AI is the main reason crawler impersonation has grown, because demand for training data drives scraping. He said site owners should verify crawlers with reverse DNS and IP checks rather than trusting the user-agent string.

How does Google say to verify Googlebot?

Google's verification page describes a reverse DNS lookup on the IP, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, and a forward DNS lookup that must return the same IP. For large-scale checks it recommends matching IPs against its published IP range files.

Does reverse DNS work for every Google crawler?

Not in our test. All 40 sampled Googlebot-range addresses verified, but 31 of 40 sampled addresses in Google's Google Cloud user-triggered fetcher ranges had no reverse DNS record. Those addresses can only be confirmed with the IP range files.

Is Web Bot Auth a replacement for IP and DNS checks?

Not yet. Google's documentation calls its Web Bot Auth support experimental, says not all Google user agents use it and not every request is signed, and tells site owners to keep using IP addresses, reverse DNS and user-agent strings.

###