robots.txt
All of our reporting and guides on robots.txt.
- Guides
Robots.txt for AI Crawlers: Which Bots to Allow, and Why
Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.
