robots.txt
All of our reporting and guides on robots.txt.
- Google
Facebook Blocks Google-Extended. Gabe Sees It in Gemini
Facebook's robots.txt bars Google-Extended, the token Google ties to Gemini grounding. Glenn Gabe's Semrush screenshots show Gemini visibility anyway.
- Guides
Shared AI Opt-Out Misses Bots a Viral List Blocks
A shared robots.txt signal for AI training reaches only the companies that honor it, and it runs against the training bots a viral list says to keep.
- News
Discord's Search Rollout Leaves AI Bots Unblocked
Discord says public-channel messages reach web search in December 2026. Its robots.txt names no AI crawler, and brands should watch AI answers.
- News
News Sites Block Meta's Trainer More Than Its Search Bot
Publishers say Meta's revised terms still allow AI answers, but a robots.txt check of 24 news sites finds 10 block Meta's training bot by name and only four its AI search bot.
- News
None of 23 News Sites Block Meta's Ads Crawler by Name
Meta says robots.txt still works, but none of 23 news sites block Meta-ExternalAds by name, the crawler whose docs echo the new terms.
- News
12 of 20 News Sites Block Meta's AI Training Crawler
Twelve of 20 sampled news sites block Meta's AI training crawler, and only four block the one tied to citations. Meta's October 30 terms may redraw that line.
- Guides
Robots.txt for AI Crawlers: Which Bots to Allow, and Why
Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.






