Saturday, November 21, 2026

Reporting and field guides on AI search, AEO, and GEO

Topic

AI crawlers

All of our reporting and guides on AI crawlers.

  1. Guides

    SearchBot Is the Floor. ChatGPT-User Is the Signal

    OAI-SearchBot crawls show a page was fetched while ChatGPT-User fetches show a real question pulled it in, and page cache hides crawler visits from WordPress counts.

  2. Research

    AI Mode's Reported Fetcher Has No 'Bot' in Its Name

    Dave Smart reports AI Mode fetches as Google-Lens and Gemini as Google. Neither has bot, crawl or spider in it. A geo redirect rule could misroute them. Our check found no edge blocking.

  3. News

    NYT Sues Brave Over a Crawler No One Can Block by Name

    The Times says Brave gives site owners no way to block its crawler, and Brave's own page says its bot follows Googlebot's access rules.

  4. Guides

    Shared AI Opt-Out Misses Bots a Viral List Blocks

    A shared robots.txt signal for AI training reaches only the companies that honor it, and it runs against the training bots a viral list says to keep.

  5. News

    Discord's Search Rollout Leaves AI Bots Unblocked

    Discord says public-channel messages reach web search in December 2026. Its robots.txt names no AI crawler, and brands should watch AI answers.

  6. Google

    ChatGPT Sites Invite Bots After Google's Noindex Advice

    Google told hosts to keep shared pages out of search by default, and one ChatGPT Sites business page invites Googlebot and AI crawlers in.

  7. News

    News Sites Block Meta's Trainer More Than Its Search Bot

    Publishers say Meta's revised terms still allow AI answers, but a robots.txt check of 24 news sites finds 10 block Meta's training bot by name and only four its AI search bot.

  8. Platforms

    Genezio Says OpenAI's Search Bot Runs JavaScript

    Genezio says OpenAI's search crawler ran its JavaScript, but TranX found ChatGPT's page fetcher reads only HTML, leaving JavaScript-only sites a gamble.

  9. News

    None of 23 News Sites Block Meta's Ads Crawler by Name

    Meta says robots.txt still works, but none of 23 news sites block Meta-ExternalAds by name, the crawler whose docs echo the new terms.

  10. News

    12 of 20 News Sites Block Meta's AI Training Crawler

    Twelve of 20 sampled news sites block Meta's AI training crawler, and only four block the one tied to citations. Meta's October 30 terms may redraw that line.

  11. Platforms

    Profound's Bot Rendering Raises a Gemini Policy Question

    Profound's new bot rendering serves AI crawlers their own copy of a page, and its planned bot-specific versions collide with Google's crawler rules.

  12. Platforms

    Cloudflare Adds a Web Search API to AI Gateway

    Developers can now pull live web results into model calls through Cloudflare, with Ceramic.ai, Exa, and Linkup as launch partners. The company is pairing it with rules for how search crawlers should behave.

  13. Guides

    Robots.txt for AI Crawlers: Which Bots to Allow, and Why

    Blocking the wrong user agent can quietly remove you from ChatGPT, Claude, or Perplexity answers. A field guide to the main AI crawlers, what each one does, and a sensible default.