Shared AI Opt-Out Misses Bots a Viral List Blocks

You've heard that one new robots.txt line handles AI training. You've also seen a viral list that tells you to keep the training bots in. Both can't be your plan.

On October 30, 2026, OpenAI said it will honor a shared robots.txt signal for AI training preferences. On November 1, 2026, Ilias Ism posted a keep-or-block list that keeps GPTBot and ClaudeBot, the training crawlers, and blocks CCBot, Bytespider and Meta's crawlers.

The signal comes from AI developers and says nothing about the crawlers on his block list. It also runs against his reason for keeping the two training bots. My read is that training access and crawler access are separate decisions and the file should say so.

The signal came first, on a Friday.

OpenAI wrote that "OpenAI will honor a common signal in robots.txt" for generative AI training preferences. The company called it a voluntary effort meant to reach several participating providers at once. Search Engine Journal reports that Google and Microsoft plan to honor it too. On the timing, OpenAI wrote only "We aim to implement the signal in the coming months."

The signal comes from an Internet-Draft by Kevin Kelley of Anthropic, which the IETF says has no formal standing yet. As the draft proposes it, a site adds a robots.txt group that names the token __usage_ai-training and disallows the paths it wants kept out of training. A User-Agent: * group does not count, and a file with no group for the token expresses no preference at all.

The draft also says the token "is not a directive for crawling behavior" and calls usage preferences and crawler rules "independent and orthogonal." OpenAI says the same from its side. The new signal "will not replace the controls we already provide through robots.txt." The per-bot rules for GPTBot and OAI-SearchBot stay in force. As Unite.AI summarizes OpenAI's crawler documentation, each setting is independent. A site can appear in ChatGPT search while disallowing GPTBot.

Then came the list

Marie Haynes, an SEO consultant, wrote on X on October 31 that the new control means you don't have to block bots individually. The next day Ism posted his list. There is no sign he wrote it in answer to the announcement.

He says it comes from 10 million requests across 22 of his sites, and that he has not published how they were counted. He writes that 45 percent of web traffic is bots now, and that blocking every AI bot kills your ChatGPT and Perplexity citations.

His keep list covers the search and answer bots, including OAI-SearchBot, Claude-SearchBot and PerplexityBot. It also keeps GPTBot and ClaudeBot. Ism calls them training crawlers rather than live search, and says you want future models to know your brand before they ever search.

His block list runs from Bytespider and CCBot to meta-externalagent and meta-externalads. He says those crawlers made more than 1.6 million requests a week on his sites and sent almost no clicks back.

Where they collide

Two gaps show up. The first is scope. The signal binds the companies that choose to honor it, and nothing in OpenAI's post or the draft adds CCBot, Bytespider or Meta's crawlers to that group. Those need their own named groups, whatever the signal does.

The second is intent. One file can hold the token and an open door for GPTBot, because the draft keeps crawling and usage apart. But Ism keeps GPTBot and ClaudeBot for a stated reason, which is that he wants models trained on his brand. The token is how a site says it does not want that. Adopting both means contradicting yourself on purpose.

Enforcement is the third problem. Asmir, a commenter on Ism's thread, argued that the scrapers worth worrying about mostly ignore robots.txt, and that you need a WAF rule to really block them. I can't verify that from what is published. OpenAI does call its signal voluntary, and a named Disallow group works the same way.

What I make of it

Ism's list is a reasonable answer to a traffic question. Search and answer bots can send people, and I understand wanting to drop crawlers that send none. I'd go further than the click count, though. A crawler that sends no clicks can still feed the conversations that make someone search your name later, and a referral log won't show that. His numbers measure clicks, and clicks were never the only reason a brand gets named.

The token is a different question. It asks whether a company may train on your pages, and it only works on companies that read it. The answer to Ism is two decisions. Decide whether you want your content used for training, and decide separately which bots you let retrieve it. A list that merges them hands you one answer for both, and you may not have chosen it.