Blocking Meta's AI training bot does nothing about the bot Meta says helps it cite and link to your pages in Meta AI.

I checked 24 US and UK news sites on October 25, 2026. Ten block Meta-ExternalAgent, which Meta says it uses for training foundation models, by name. Only four block Meta-WebIndexer by name, the bot behind Meta AI search.

Publishers are fighting Meta over AI answers in a contract, as PPC Land reported. I think the files tell a different, and less settled, story.

The European Publishers Council opened the contract fight.

Meta's new Commercial Terms and Self-Serve Ad Terms let it crawl websites "associated with" a business's accounts, according to Nieman Lab. On October 16, 2026, Meta removed the word train. The clause now limits the purpose to providing, developing, improving, researching and maintaining "advertising and business products," plus the integrity and security of Meta's products, PPC Land reports. On October 23, 2026, Meta pushed the start date from October 30 to November 20.

A Meta spokesperson told Nieman Lab the extra time is to explain how Meta uses web crawlers and "the fact that content owners will still be able to use robots.txt to signal preferences." Nieman Lab reports that "No versions of the terms include language about respecting robots.txt preferences." I read Meta's live terms preview on October 25 and found the same. It does not mention robots.txt at all.

The European Publishers Council, a Brussels trade body for news publishers, wrote a second time to the European Commission on October 23, 2026. Its executive director, Angela Mills Wade, posted the case on LinkedIn on October 24, as quoted by PPC Land. "Meta has removed the explicit reference to AI training but its terms still seek sweeping permissions to crawl, collect, analyse and reuse content from businesses' websites. Crucially, they do not expressly exclude AI grounding or the generation of AI answers and summaries."

The council also rejects Meta's robots.txt answer. "Technical controls over crawler access cannot substitute for clear, enforceable contractual limits on collection and subsequent reuse," the council said in its October 23 statement. PPC Land notes that the clause names no crawler.

Legal experts doubt the delay too. Thomas Höppner, a professor and attorney specializing in competition law in Germany, told Nieman Lab: "My fear is that this is merely a measure by Meta to reduce the risk of court actions for interim measures." James Grimmelmann, a professor of digital and information law at Cornell Law School, was less sure what the delay means. "I don't think that repeating the same non-explanations for another three weeks will make much difference," he said. "But maybe they'll use the time to come up with better explanations."

Meta's bots do different jobs

Meta and the council both discuss robots.txt without naming a bot. Meta's own crawler page, updated May 21, 2026, splits it by bot.

Meta-WebIndexer navigates the web to improve Meta AI search quality. The page says "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." Meta-ExternalAds crawls for "improving advertising and other business-related products and services." Meta-ExternalFetcher fetches links at a user's request, and Meta says it "may bypass robots.txt rules."

Meta's terms name no bot for the clause. The purpose in the revised text is advertising and business products, which sounds like Meta-ExternalAds. That is my read, not Meta's statement.

My check ran a script against the robots.txt files of 24 readable news sites on October 25 at 07:25 UTC. A 25th site, politico.com, returned a 403. The script counts only groups that name a Meta bot. Meta-WebIndexer is blocked by name at four sites: nytimes.com, usatoday.com, newyorker.com and nbcnews.com, which leaves a few paths open. Meta-ExternalAgent is blocked by name at 10, and Meta-ExternalFetcher at eight. No site blocks Meta-ExternalAds by name.

Bar chart of how many of 24 news sites block each Meta crawler by name in robots.txt: 10 for ExternalAgent, 8 for ExternalFetcher, 4 for WebIndexer
The training bot is named far more often than the bot Meta says helps it cite pages in Meta AI. No site names Meta-ExternalAds.

Two sites went the other way. Reuters names Meta-WebIndexer and Meta-ExternalAds in the group of bots it lets in. The Wall Street Journal blocks every bot it does not name with a wildcard rule, and it names neither. A robots.txt file cannot show firewall or CDN rules. These counts describe the files and nothing deeper.

What the files do and do not say

Some of this looks like a decision. The New York Times blocks Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher by name, and Reuters named the indexer on purpose. For the other sites I cannot tell. A missing rule can mean a choice to stay open, or it can mean nobody has looked.

I see two separate decisions here, the contract and the file, and publishers are making only one of them loudly. The council may be right that a file cannot replace a clause. A file is still where a site says which Meta bot may read it, and allowing the indexer only helps Meta cite a page. It promises no citation. Whether that trade is worth it is a question each newsroom has to answer for itself, not one the training-bot block answers for them.

Free newsletter

The Sherpa Brief

One email a week on what changed in AI search, what it means, and what to do about it. Written for marketers, SEOs, and publishers.

Free. Unsubscribe in one click.