You added a Meta-ExternalAgent rule to your robots.txt, and you figure Meta's new crawling terms can't reach you.

Read the terms next to Meta's crawler docs and that rule may be aimed at the wrong bot. On October 18, 2026, AEO Sherpa parsed 23 news-site robots.txt files. Thirteen block Meta-ExternalAgent by name. None block Meta-ExternalAds by name, the crawler the docs describe in language close to the revised clause.

Meta says robots.txt still works. I think that is true, and it still leaves most of these files silent on the bot in question.

Both Meta's revision and its robots.txt promise followed Nieman Lab's reporting.

Nieman Lab reported on October 15 that Meta's new Commercial Terms, effective October 30, would let it crawl any domain "associated with your products or services" for a business account. The original wording said Meta "may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain" its products.

On October 17, Meta revised the clause. It dropped that sentence and the word train, and it narrowed the purpose. The current terms allow the crawling "to provide, develop, improve, research, and maintain advertising and business products" and to maintain the integrity and security of Meta's products.

A Meta spokesperson told Nieman Lab that "News sites, like all other content owners, still have the ability to disable the crawler via robots.txt flags. That has been the case, and it has not changed." The terms themselves name no crawler and never mention robots.txt.

The docs describe a different bot

Meta's crawler documentation, updated May 21, 2026, lists its crawlers and what each is used for. Meta-ExternalAds "crawls the web for use cases such as improving advertising and other business-related products and services." Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly."

The first description sounds like the revised clause. That match is my comparison of the two texts. Meta's terms don't say which crawler would do the work.

AEO Sherpa fetched robots.txt from 25 news sites on October 18 at about 15:28 UTC and parsed 23 of them. Axios and Politico returned 403.

The named blocks include The New York Times, The Washington Post and the BBC for Meta-ExternalAgent. WSJ and Reuters shut it out through catch-all rules. For Meta-ExternalAds the picture differs. Reuters names it in a group of allowed bots. WSJ stops it only through a catch-all rule that shuts out every bot it does not list. The other 21 never mention it, and eight of the 23 have no rules for any Meta crawler. CNN, AP and NPR are among them.

Bar chart: 13 of 23 news sites block Meta-ExternalAgent by name, 9 block Meta-ExternalFetcher, 0 block Meta-ExternalAds
A Meta-ExternalAgent rule leaves Meta-ExternalAds untouched.

Nine files block Meta-ExternalFetcher by name, the bot Meta says may bypass robots.txt because it fetches links at a user's request.

Is robots.txt enough?

The lawyers Nieman Lab consulted make two separate points. The first is about how far the new wording reaches. Matt Topic, a media and intellectual property attorney at Loevy & Loevy, said "The revised version of the new terms still allows Meta to use that scraped content to develop any 'business product.' That doesn't seem to be a defined term, and presumably Meta is developing AI business products. I think 'develop' is broad enough to include 'train.'"

James Grimmelmann, who teaches digital and information law at Cornell Law School, told Nieman Lab "'develop...advertising and business products' [language] is still broad enough to drive a truck through."

Benjamin Sobel, a law professor at the University of Wisconsin, told Nieman Lab that robots.txt states a preference and nobody has to comply.

There is also a second gap in the documentation. The revised clause cites "integrity and security," and Meta's docs say its FacebookExternalHit crawler "might bypass robots.txt when performing security or integrity checks." Both texts use the same words, and I read that as a reason to look closer.

I think the fight over wording is the loud part of this story. The audit shows something quieter. A rule that blocks Meta-ExternalAgent does not touch Meta-ExternalAds, and in these 23 files the second bot has almost no rules at all. Meta has not named the crawler behind the new terms, and a robots.txt that covers only ExternalAgent leaves that question open. Our October 16 piece counted only Meta-ExternalAgent blocks, and this is the wider rerun.

Free newsletter

The Sherpa Brief

One email a week on what changed in AI search, what it means, and what to do about it. Written for marketers, SEOs, and publishers.

Free. Unsubscribe in one click.