Saturday, November 21, 2026

Reporting and field guides on AI search, AEO, and GEO

Topic

robots.txt

All of our reporting and guides on robots.txt.

  1. Google

    Facebook Blocks Google-Extended. Gabe Sees It in Gemini

    Facebook's robots.txt bars Google-Extended, the token Google ties to Gemini grounding. Glenn Gabe's Semrush screenshots show Gemini visibility anyway.

  2. Guides

    Shared AI Opt-Out Misses Bots a Viral List Blocks

    A shared robots.txt signal for AI training reaches only the companies that honor it, and it runs against the training bots a viral list says to keep.

  3. News

    Discord's Search Rollout Leaves AI Bots Unblocked

    Discord says public-channel messages reach web search in December 2026. Its robots.txt names no AI crawler, and brands should watch AI answers.

  4. News

    News Sites Block Meta's Trainer More Than Its Search Bot

    Publishers say Meta's revised terms still allow AI answers, but a robots.txt check of 24 news sites finds 10 block Meta's training bot by name and only four its AI search bot.

  5. News

    None of 23 News Sites Block Meta's Ads Crawler by Name

    Meta says robots.txt still works, but none of 23 news sites block Meta-ExternalAds by name, the crawler whose docs echo the new terms.

  6. News

    12 of 20 News Sites Block Meta's AI Training Crawler

    Twelve of 20 sampled news sites block Meta's AI training crawler, and only four block the one tied to citations. Meta's October 30 terms may redraw that line.

  7. Guides

    Robots.txt for AI Crawlers: Which Bots to Allow, and Why

    Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.