You added a Meta-ExternalAgent line to robots.txt and figured the question was closed. On October 30, 2026 a contract reopens it.

I read the live robots.txt files of 20 major US and UK news sites on October 16. Twelve of those 20 block Meta's AI training crawler. Only four block Meta-WebIndexer, the crawler Meta says helps it cite sites in Meta AI. Training access and citation access are two separate choices, and publishers are making them differently.

Meta's new commercial terms take effect that day. They let Meta crawl the domains associated with a business's products or services and train on what it finds, and Nieman Lab reports there is no opt-out before then. A robots.txt line asks Meta to stay out. The contract says Meta may come in. My read is that the contract is the stronger document.

Nieman Lab broke the story on October 15, after a reader tip.

The clause lets Meta use automated software, and "test credentials," to crawl the domains and pages associated with a business's products or services. Meta may then use "any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products." The terms count a managed Page, an ad campaign or a Meta Business Suite login as business use. Reporters Andrew Deck and Hanaa' Tameez note that virtually every news organization has at least one such account, Nieman Lab included.

The exit is to leave. The terms say "If you do not agree to the updated terms, you may stop all access or use of our Products for business or commercial purposes." Continuing to use them after notice counts as agreeing.

A Meta spokesperson told Nieman Lab the update only consolidates crawling language that was scattered across several terms and that "content owners can still manage how their content is accessed by using robots.txt."

Jason Kint, CEO of Digital Content Next, answered that on X on October 16. "Then why demand sweeping permissions in a binding contract?" he wrote.

The legal experts Nieman Lab consulted lean his way. James Grimmelmann, a professor of digital and information law at Cornell Law School, said "For news publishers, this clearly represents permission to Meta to train AI on any content from their sites." Matt Topic, a media and intellectual property attorney at Loevy & Loevy, was more careful. "It's not clear whether a court would enforce this, but publishers should assume that it would be enforceable."

Nieman Lab also reports that the experts see an opening for Meta to argue its commercial terms outrank other blocking instructions. The full terms preview never mentions robots.txt. Whether the contract beats a disallow is an interpretation, and nobody has tested it.

Which Meta bot you are actually blocking

Meta's crawler documentation, last updated May 21, 2026, splits the work between several bots. Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." Meta-WebIndexer is the search side. Meta says "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." Meta-ExternalFetcher grabs pages at a user's request, and Meta says it "may bypass robots.txt rules." FacebookExternalHit builds link previews for shared pages and might also bypass robots.txt for security or integrity checks.

So I checked who blocks what. On October 16 at about 15:30 UTC I fetched the live robots.txt of 20 news sites and read each file with Python's robotparser module. Ten sites name Meta-ExternalAgent and shut it out of everything. Those are The New York Times, The Washington Post, USA Today, The Guardian, the BBC, NBC News, People, The Verge, Nieman Lab and Bloomberg. Reuters and The Wall Street Journal get to the same place through a catch-all rule that bars every crawler they have not named. That makes 12 of the 20 sites.

Far fewer block the search bot. Only four sites block Meta-WebIndexer. The Times, USA Today and People name it, and the Journal's catch-all covers it. Reuters names it in a group that its file leaves open. Eight sites block neither bot: CNN, Fox News, the Associated Press, NPR, the Los Angeles Times, Axios, Business Insider and Forbes.

Column chart: 12 of 20 news sites block Meta-ExternalAgent, 4 of 20 block Meta-WebIndexer
Blocking the training crawler is common among these publishers. Blocking the search crawler is not.

These are 20 well-known publishers that I picked, which makes this a snapshot of big names and a poor guide to the whole web. It measures what each file permits. A firewall could still block a crawler that a robots.txt file allows, and I did not test that.

The line I would not rely on

The terms do not tell a publisher to drop a Meta-ExternalAgent block, and Meta says robots.txt still works. What they add is a clause that applies once a business keeps using Meta's products after October 30. That is a different kind of document than a request in a text file.

I would keep the training block. The WebIndexer decision is a separate one, and this audit cannot settle it. It shows permissions, not citations or recommendations, and citations are a diagnostic rather than the outcome I care about. Still, allowing Meta-WebIndexer is the one setting where robots.txt and the contract point the same way, and I think publishers who block it should know they chose that.

Free newsletter

The Sherpa Brief

One email a week on what changed in AI search, what it means, and what to do about it. Written for marketers, SEOs, and publishers.

Free. Unsubscribe in one click.