Cloudflare Pilot Helps OpenAI Find Millions of New Pages

You've been told the fix for missing ChatGPT citations is better content. Maybe. But if ChatGPT's crawler never found the page, the writing never got a vote.

Cloudflare and OpenAI report that a pilot covering about 1 percent of Cloudflare's network has enabled OpenAI to discover and index hundreds of millions of new web pages. Cloudflare passes network signals to OpenAI's search crawler, and OpenAI decides on its own how those pages rank.

That is another way into ChatGPT's index, next to links, sitemaps and IndexNow. These are the two companies' own results, and Cloudflare sells the network services the signals come from. My read is that a missing citation can be a discovery gap before it is a content gap.

Cloudflare says it launched the pilot with OpenAI in July and published the first results on October 15, 2026. Two OpenAI staff, Henry Aspegren and James Balfour, are listed among the post's authors.

The mechanism is short. Cloudflare's network provides signals that can help OpenAI's search crawler discover more useful pages. OpenAI then independently evaluates and ranks how they appear. Cloudflare's X account said the pilot shows "how real-time network signals can make AI search drastically smarter, fresher, and safer without overloading website infrastructure."

The signals do two jobs. They keep the crawler away from pages that don't exist or that redirect elsewhere, and they tell it when a page has changed. Cloudflare says that cuts wasted fetches. It also admits that "finding new pages can initially increase crawling as those pages are evaluated and indexed."

Cloudflare frames this as a crawl-load fix as much as a reach fix. Fetching every page more often "would be expensive for everyone," and a crawler re-downloading an unchanged page adds load for the site owner with no benefit. For Cloudflare customers, the aim is less unnecessary load.

Its example is the kind of page it wants found. The web is full of specifics, it says, like a washing machine manual for one exact model, and "these pages can be valuable even if few other sites link to them." Sparsely linked pages like that are what an alternative discovery route could help surface.

Permissions stay with site owners, Cloudflare says. "A signal about a page doesn't grant permission to access it," and the research uses metadata about the web "rather than sharing page content for model training." The one control the post names is Cloudflare's AI Crawl Control, for managing how AI systems access a site. That is an access setting, not a way into the pilot.

The results are one number and one outcome: a cohort of about 1 percent of the network, and hundreds of millions of new pages. Cloudflare names no cohort members, offers no way to join and reports nothing yet on citations or traffic. Cloudflare's next phase will "measure what happens after content is discovered."

Glen Allsopp, an SEO publisher, relayed the headline number on X and called it "Super interesting." He added that it is "only a pilot for now." Other replies were sharper. Jason Fleagle asked how quickly a deleted page stops appearing in answers, and a user called CC asked how this avoids becoming a free enrichment feed for a crawler. Both are fair questions. Finding new pages is one half of freshness, and pulling down withdrawn ones is the other. The post covers only the first half.

Discovery, retrieval, citation and recommendation are different events. This pilot measures only the first. A page that gets indexed can still go uncited. Still, a site that sits outside this lane has one fewer way to be found, and a site inside it has one more.

My guess is that the first winners will be small sites with good pages that nobody links to. That is a hypothesis, not something the pilot showed. If you are missing from ChatGPT answers, I would check whether its crawler can reach you before I rewrote a word.

Disclosure: AEO Sherpa runs on Cloudflare's EmDash CMS. That has nothing to do with this pilot.