Cloudflare Will Score Verified Bots on Robots.txt and 429 Compliance, With Trust Bands Coming to Radar

Cloudflare said on 16 October that it now measures whether Verified bots honour robots.txt and slow down after HTTP 429 responses. The results will form a trust band published on Cloudflare Radar, and customers will later be able to set a trust threshold for how Verified bot traffic is handled.

Vibhu Dhariwal
Cofounder & CTO, Digital Gratified
Published
Updated
Section
Industry news
Reading time
4 minutes
Data card: Cloudflare will score Verified bots on robots.txt and HTTP 429 compliance

Cloudflare will start rating its Verified bots on how they behave after approval, not only on who operates them. In a blog post on Verified bots published on 16 October 2026, Cloudflare's Chris Tsoukaladelis said the company has built a system that measures whether Verified bots follow robots.txt and whether they slow down when a site returns HTTP 429 (Too Many Requests). The observations will feed an "explainable trust band" for each bot that Cloudflare plans to publish on Radar.

The change affects any crawler on Cloudflare's Verified list, including search engine and AI crawlers, and gives site owners a public measure of whether those bots respect the rules they publish.

What changed

  • Behaviour now counts toward verification. Cloudflare says a Verified bot must identify itself accurately, do only what it says it will do and avoid abusive behaviour. The new system targets the third rule.
  • Two initial signals. Whether a bot's traffic is consistent with a site's published robots.txt over time, and whether it slows down or pauses after receiving 429 responses.
  • Patterns, not one-offs. Cloudflare says isolated violations may not be significant; it looks for repeated behaviour that disregards site preferences.
  • A trust band, not a verdict. The band will show how consistently a bot respects site preferences and which observed behaviours informed the rating.
  • Customer thresholds later. Cloudflare is using the system internally now and plans to let all customers choose the trust threshold used to decide how Verified bot traffic is handled.
SignalWhat Cloudflare checksEffect on trust
robots.txtWhether observed traffic is consistent with the site's published rules over time, in line with RFC 9309Repeated disregard lowers trust
HTTP 429Whether the bot slows down, pauses or keeps the same request rate after a 429Slowing or pausing raises trust; ignoring the signal lowers it
The two initial behaviour signals in Cloudflare's Verified bot trust system. Source: Cloudflare blog, 16 October 2026.

Context

Cloudflare frames the change as the difference between risk, which applies to a single request, and trust, which is built over repeated behaviour. It points to its 2025 decision to de-list Perplexity's bot from the Verified list after, in Cloudflare's account, the bot tried to reach sites that had opted out of scraping and attempted to get around blocks.

The company also says robots.txt sets minimum rules and that "there is no implication that anything not stated is allowed", meaning a bot should also stop when a site signals through other means that access is unwelcome. The robots.txt standard itself is the IETF's RFC 9309.

Identity is handled separately. Cloudflare says bots sign requests with Web Bot Auth, which its BotBase catalogues in an open format. The new trust system sits alongside Bot Preference Sync, which publishes site preferences, and Bot Management, which enforces them, in what Cloudflare calls an accountability loop.

The 429 signal mirrors how Google describes its own crawlers. Google's crawl rate documentation tells site owners to return 500, 503 or 429 to urgently reduce crawling for a short period, and says Google's crawling infrastructure then reduces the crawl rate across the hostname. Google also made recent Retry-After additions to that page.

What is not yet known

Cloudflare has not said how many bands there will be, how long the observation period is, what thresholds will apply, how a bot can recover a lower rating or when scores will appear on Radar. It also has not said how customers will configure thresholds.

What this means for SEO teams

  1. Make robots.txt precise. If Cloudflare measures compliance against your published rules, vague or outdated rules weaken the signal. Name the AI and search agents you want to allow or block. Our analysis of 514 top sites found few use machine-readable AI usage preferences in robots.txt.
  2. Use 429 deliberately. A 429 is now both a load control and evidence about a bot's conduct. Return it for real overload, and pair it with Retry-After where your stack allows.
  3. Audit your own crawlers. Teams that operate a crawler on Cloudflare's Verified list should check that it respects robots.txt and backs off on 429, since repeated violations will lower its trust band.
  4. Watch Radar. Once scores are public, they will give site owners a third-party view of which crawlers honour preferences before deciding what to allow.

Conclusion

Cloudflare is moving Verified bot status from a one-time check to a continuing record of conduct, starting with robots.txt compliance and response to HTTP 429. For site owners, it raises the practical value of clear robots.txt rules; for anyone operating a crawler, it means behaviour on other people's sites will be visible and, eventually, used to decide whether the crawler gets through.

Questions and answers

What is Cloudflare changing for Verified bots?

Cloudflare said on 16 October 2026 that it is measuring whether Verified bots respect robots.txt and slow down after HTTP 429 responses, and will turn those observations into a trust band for each bot.

Will the trust scores be public?

Cloudflare says it plans to share the scores publicly on Cloudflare Radar. It has not given a date.

Can site owners act on the trust band?

Not yet. Cloudflare says it plans to make the system available to all customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled.

Does one robots.txt violation count against a bot?

Cloudflare says isolated violations may not be significant and that it looks for repeated patterns of behaviour that disregard site preferences.

More releases

  1. Studies

    Do Title Tag Keywords Still Rank? Only 27.5% of 262 Google Titles Held the Exact Query

    Google says titles are a small ranking factor, but our pull of 262 US Google results for 30 queries found only 27.5% of title links held the exact query, and first place rarely had one. DOJ notes and leaked Google documents explain why titles still count.

  2. Updates

    Search Console AI Report Bug Hit Core Update Launch; 3 of 8 2026 Updates Overlapped

    Google has logged an error that cut impressions in Search Console's Generative AI performance report for 9 to 11 November, the window in which the November 2026 core update began. Our timeline of Google's own records shows three of this year's eight ranking updates launched on a day with a logged data error.

  3. Explainers

    Bing's AI-Off Extension Just Adds noAI=1 to Searches; It Has 137 Chrome Users

    Bing documented two ways to hide its AI answers on 11 November: a preview browser extension and a -ai query operator. Our read of the extension's code shows it appends a noAI=1 parameter, ships with AI off, and resets the default search engine and new tab to Bing. It has 137 Chrome users.

Every release

Tell us the site and the goals.

Rankings in Google and visibility in AI answers, from one monthly plan. One call is enough to say which one fits. Email works just as well.