Cloudflare Signals Helped OpenAI Index Hundreds of Millions of New Pages From About 1% of Its Network

Ras Al Khaimah, United Arab Emirates, 18 October 2026.

Cloudflare and OpenAI published early results on 15 October 2026 from a research pilot that feeds signals from Cloudflare's network to OpenAI's search crawler. According to the joint post on the Cloudflare blog, written by two Cloudflare staff and two OpenAI staff, the pilot used a participating cohort of about 1% of Cloudflare's network and has helped OpenAI discover and index hundreds of millions of new web pages for ChatGPT.

The next day Cloudflare announced a separate system that will grade Verified bots on whether they keep obeying site rules. Taken together, the two posts show a CDN positioning itself as both a discovery channel and a referee for AI crawlers.

What the pilot found

• Start date and scale. The pilot began in July 2026. Cloudflare describes the cohort as roughly 1% of its total network.
• Result. Pages that OpenAI's index did not previously know about were brought in, which Cloudflare says gives ChatGPT more sources for timely answers. The post gives no exact page count beyond "hundreds of millions".
• Division of labour. Cloudflare supplies discovery and change signals; OpenAI says it independently evaluates and ranks how those pages appear.
• Waste filtering. The partners say automated traffic can make nonexistent pages look popular, that some sites redirect almost any address to the homepage, and that duplicate copies are common. Spotting those patterns lets a crawler skip fetches that help nobody.
• Change signals. Signals that a page has changed tell the crawler when a revisit is worthwhile, to avoid repeatedly downloading unchanged pages. The post admits that newly found pages can first increase crawling while they are evaluated.
• Safety flags. The partners are testing page-level signals for harmful content, such as a single planted fake login form, so one page can be kept out of an index without penalising the whole domain.

Controls stay with site owners, Cloudflare says

The post says a signal about a page does not grant permission to access it, and that the research uses metadata about the web for search and discovery rather than sharing page content for model training. Site owners can manage AI access through Cloudflare's AI Crawl Control.

OpenAI's own crawler documentation treats search and training as separate choices: a site can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot to keep content out of foundation model training.

A day later: trust bands for Verified bots

On 16 October Cloudflare published Continuous accountability for Verified bots. Being on Cloudflare's Verified list has meant proving who operates a bot; the new system adds an ongoing check on how the bot behaves.

Signal	What Cloudflare measures	Effect on trust
robots.txt	Whether a bot's traffic matches a site's published preferences over time, in line with RFC 9309	Repeated patterns of disregard lower trust; isolated one-offs may not count
HTTP 429	Whether a bot slows down or pauses after a Too Many Requests response	Slowing or pausing raises trust; ignoring the signal lowers it

Cloudflare says it is already using the system internally to build behavioural history. It plans to publish scores on Radar and to let all customers choose a trust threshold that decides how Verified bot traffic is handled. It gave no dates for either step. The post recalls that Cloudflare removed Perplexity's bot from its Verified list in 2025 after it tried to reach sites that had opted out and then tried to get around blocks. Cloudflare also notes that robots.txt sets minimum rules: anything it does not mention is not automatically allowed. The RFC 9309 standard defines how robots.txt rules are parsed, and RFC 6585 defines the 429 status code.

What this means for SEO teams

1. Separate search from training in robots.txt. If you want ChatGPT search visibility but not training use, allow OAI-SearchBot and disallow GPTBot, as OpenAI's documentation describes.
2. Clean up soft 404s and catch-all redirects. The pilot specifically flags sites that redirect almost any URL to the homepage. Return real 404 or 410 codes so discovery systems do not waste effort or misread your site.
3. Make change dates reliable. The partners say change signals decide when a page is worth another visit, so keep last-modified values in sitemaps and headers accurate.
4. Use 429 deliberately. If AI crawlers overload your origin, a 429 response is now a signal that Cloudflare will use to judge whether Verified bots behave.
5. Check for planted pages. Page-level harm flags are being tested, so scan for injected login or spam pages that could be excluded from AI indexes.

Conclusion

The pilot suggests that AI search indexes are still missing a large share of the web, and that network-level signals can close part of that gap quickly. For SEOs it is a reminder that discoverability in ChatGPT depends on crawl permissions, clean status codes and trustworthy change signals, not just on links. Cloudflare's trust bands add a second message: the same company that helps crawlers find pages now intends to publish how well those crawlers respect site owners.

Questions and answers

What did the Cloudflare and OpenAI pilot do?

According to the Cloudflare blog post of 15 October 2026, network signals from Cloudflare helped OpenAI's search crawler discover pages it did not know about. OpenAI then evaluated and ranked those pages independently.

How large was the pilot?

Cloudflare says the participating cohort was about 1% of its total network, and that the pilot helped OpenAI discover and index hundreds of millions of new web pages.

Does the pilot share page content with OpenAI for training?

No, according to Cloudflare. It says the research uses metadata about the web for search and discovery, not page content for model training, and that a signal about a page does not grant permission to access it.

What are Cloudflare's Verified bot trust bands?

Announced on 16 October 2026, they are ranges that describe how consistently a Verified bot respects robots.txt and slows down after HTTP 429 responses. Cloudflare plans to publish them on Radar and let customers set trust thresholds.

###