Cloudflare Will Score Verified Bots on Robots.txt and 429 Compliance, With Trust Bands Coming to Radar

Ras Al Khaimah, United Arab Emirates, 19 October 2026.

Cloudflare will start rating its Verified bots on how they behave after approval, not only on who operates them. In a blog post published on 16 October 2026, Cloudflare's Chris Tsoukaladelis said the company has built a system that measures whether Verified bots follow robots.txt and whether they slow down when a site returns HTTP 429 (Too Many Requests). The observations will feed an "explainable trust band" for each bot that Cloudflare plans to publish on Radar.

The change affects any crawler on Cloudflare's Verified list, including search engine and AI crawlers, and gives site owners a public measure of whether those bots respect the rules they publish.

What changed

• Behaviour now counts toward verification. Cloudflare says a Verified bot must identify itself accurately, do only what it says it will do and avoid abusive behaviour. The new system targets the third rule.
• Two initial signals. Whether a bot's traffic is consistent with a site's published robots.txt over time, and whether it slows down or pauses after receiving 429 responses.
• Patterns, not one-offs. Cloudflare says isolated violations may not be significant; it looks for repeated behaviour that disregards site preferences.
• A trust band, not a verdict. The band will show how consistently a bot respects site preferences and which observed behaviours informed the rating.
• Customer thresholds later. Cloudflare is using the system internally now and plans to let all customers choose the trust threshold used to decide how Verified bot traffic is handled.

Signal	What Cloudflare checks	Effect on trust
robots.txt	Whether observed traffic is consistent with the site's published rules over time, in line with RFC 9309	Repeated disregard lowers trust
HTTP 429	Whether the bot slows down, pauses or keeps the same request rate after a 429	Slowing or pausing raises trust; ignoring the signal lowers it

Context

Cloudflare frames the change as the difference between risk, which applies to a single request, and trust, which is built over repeated behaviour. It points to its 2025 decision to de-list Perplexity's bot from the Verified list after, in Cloudflare's account, the bot tried to reach sites that had opted out of scraping and attempted to get around blocks.

The company also says robots.txt sets minimum rules and that "there is no implication that anything not stated is allowed", meaning a bot should also stop when a site signals through other means that access is unwelcome. The robots.txt standard itself is the IETF's RFC 9309.

Identity is handled separately. Cloudflare says bots sign requests with Web Bot Auth, which its BotBase catalogues in an open format. The new trust system sits alongside Bot Preference Sync, which publishes site preferences, and Bot Management, which enforces them, in what Cloudflare calls an accountability loop.

The 429 signal mirrors how Google describes its own crawlers. Google's crawl rate documentation tells site owners to return 500, 503 or 429 to urgently reduce crawling for a short period, and says Google's crawling infrastructure then reduces the crawl rate across the hostname. We covered Google's recent Retry-After additions to that page.

What is not yet known

Cloudflare has not said how many bands there will be, how long the observation period is, what thresholds will apply, how a bot can recover a lower rating or when scores will appear on Radar. It also has not said how customers will configure thresholds.

What this means for SEO teams

1. Make robots.txt precise. If Cloudflare measures compliance against your published rules, vague or outdated rules weaken the signal. Name the AI and search agents you want to allow or block. Our analysis of 514 top sites found few use machine-readable AI usage preferences in robots.txt.
2. Use 429 deliberately. A 429 is now both a load control and evidence about a bot's conduct. Return it for real overload, and pair it with Retry-After where your stack allows.
3. Audit your own crawlers. Teams that operate a crawler on Cloudflare's Verified list should check that it respects robots.txt and backs off on 429, since repeated violations will lower its trust band.
4. Watch Radar. Once scores are public, they will give site owners a third-party view of which crawlers honour preferences before deciding what to allow.

Conclusion

Cloudflare is moving Verified bot status from a one-time check to a continuing record of conduct, starting with robots.txt compliance and response to HTTP 429. For site owners, it raises the practical value of clear robots.txt rules; for anyone operating a crawler, it means behaviour on other people's sites will be visible and, eventually, used to decide whether the crawler gets through.

Questions and answers

What is Cloudflare changing for Verified bots?

Cloudflare said on 16 October 2026 that it is measuring whether Verified bots respect robots.txt and slow down after HTTP 429 responses, and will turn those observations into a trust band for each bot.

Will the trust scores be public?

Cloudflare says it plans to share the scores publicly on Cloudflare Radar. It has not given a date.

Can site owners act on the trust band?

Not yet. Cloudflare says it plans to make the system available to all customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled.

Does one robots.txt violation count against a bot?

Cloudflare says isolated violations may not be significant and that it looks for repeated patterns of behaviour that disregard site preferences.

###