AI crawlers
All of our reporting and guides on AI crawlers.
- Platforms
Cloudflare Adds a Web Search API to AI Gateway
Developers can now pull live web results into model calls through Cloudflare, with Ceramic.ai, Exa, and Linkup as launch partners. The company is pairing it with rules for how search crawlers should behave.
- Guides
Robots.txt for AI Crawlers: Which Bots to Allow, and Why
Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.
