AI crawlers
All of our reporting and guides on AI crawlers.
- Guides
SearchBot Is the Floor. ChatGPT-User Is the Signal
OAI-SearchBot crawls show a page was fetched while ChatGPT-User fetches show a real question pulled it in, and page cache hides crawler visits from WordPress counts.
- Research
AI Mode's Reported Fetcher Has No 'Bot' in Its Name
Dave Smart reports AI Mode fetches as Google-Lens and Gemini as Google. Neither has bot, crawl or spider in it. A geo redirect rule could misroute them. Our check found no edge blocking.
- News
NYT Sues Brave Over a Crawler No One Can Block by Name
The Times says Brave gives site owners no way to block its crawler, and Brave's own page says its bot follows Googlebot's access rules.
- Guides
Shared AI Opt-Out Misses Bots a Viral List Blocks
A shared robots.txt signal for AI training reaches only the companies that honor it, and it runs against the training bots a viral list says to keep.
- News
Discord's Search Rollout Leaves AI Bots Unblocked
Discord says public-channel messages reach web search in December 2026. Its robots.txt names no AI crawler, and brands should watch AI answers.
- Google
ChatGPT Sites Invite Bots After Google's Noindex Advice
Google told hosts to keep shared pages out of search by default, and one ChatGPT Sites business page invites Googlebot and AI crawlers in.
- News
News Sites Block Meta's Trainer More Than Its Search Bot
Publishers say Meta's revised terms still allow AI answers, but a robots.txt check of 24 news sites finds 10 block Meta's training bot by name and only four its AI search bot.
- Platforms
Genezio Says OpenAI's Search Bot Runs JavaScript
Genezio says OpenAI's search crawler ran its JavaScript, but TranX found ChatGPT's page fetcher reads only HTML, leaving JavaScript-only sites a gamble.
- News
None of 23 News Sites Block Meta's Ads Crawler by Name
Meta says robots.txt still works, but none of 23 news sites block Meta-ExternalAds by name, the crawler whose docs echo the new terms.
- News
12 of 20 News Sites Block Meta's AI Training Crawler
Twelve of 20 sampled news sites block Meta's AI training crawler, and only four block the one tied to citations. Meta's October 30 terms may redraw that line.
- Platforms
Profound's Bot Rendering Raises a Gemini Policy Question
Profound's new bot rendering serves AI crawlers their own copy of a page, and its planned bot-specific versions collide with Google's crawler rules.
- Platforms
Cloudflare Adds a Web Search API to AI Gateway
Developers can now pull live web results into model calls through Cloudflare, with Ceramic.ai, Exa, and Linkup as launch partners. The company is pairing it with rules for how search crawlers should behave.
- Guides
Robots.txt for AI Crawlers: Which Bots to Allow, and Why
Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.












