Can AI read your website?
01
THE CRAWLERS WE CHECK, AND WHAT EACH ONE DOES
- OAI-SearchBotOpenAI · AI search
- Indexes pages for ChatGPT search results.
- ChatGPT-UserOpenAI · User fetch
- Fetches a page when a ChatGPT user's request needs it.
- GPTBotOpenAI · Training
- Collects content that may train OpenAI's models.
- PerplexityBotPerplexity · AI search
- Indexes pages for Perplexity's answers.
- Perplexity-UserPerplexity · User fetch
- Fetches a page when a Perplexity user's question needs it.
- Claude-SearchBotAnthropic · AI search
- Indexes pages for Claude's search results.
- Claude-UserAnthropic · User fetch
- Fetches a page when a Claude user's request needs it.
- ClaudeBotAnthropic · Training
- Collects content that may train Anthropic's models.
- GooglebotGoogle · AI search
- Google Search, including AI Overviews and AI Mode.
- Google-ExtendedGoogle · Training
- Gemini training and grounding. Does not affect Google Search.
- BingbotMicrosoft · AI search
- Bing search and Copilot answers.
- ApplebotApple · AI search
- Siri and Spotlight suggestions.
- Applebot-ExtendedApple · Training
- Whether Apple may use content to train its models.
- CCBotCommon Crawl · Training
- The open web crawl many AI models are trained on.
- meta-externalagentMeta · Training
- Collects content for Meta's AI models and products.
Search and user-fetch crawlers decide whether you can be named in AI answers. Training crawlers decide whether your content can train future models. They are separate choices in robots.txt.
02
QUESTIONS
- Which AI crawlers should I allow?
- If you want to be named in AI answers, allow the search and user-fetch crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User, plus Googlebot and Bingbot. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot are a separate choice: blocking them keeps your content out of model training but does not remove you from AI search.
- Does blocking GPTBot stop ChatGPT from recommending me?
- Not on its own. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training, and controls them separately in robots.txt. Blocking OAI-SearchBot is what keeps your pages out of ChatGPT's search answers.
- Does Google-Extended affect Google Search or AI Overviews?
- No. Google says Google-Extended controls whether content is used for Gemini models and grounding, and does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode use Googlebot, so a page needs to be indexed and allowed a snippet.
- My robots.txt allows everything. Why might AI still not see my site?
- Three common reasons: a CDN or firewall rule blocks AI crawlers before robots.txt is read (Cloudflare has blocked AI training crawlers by default on new domains since July 2025, and its newer AI bot policies can block more); the page copy only appears after JavaScript runs, which AI crawlers do not execute; or a noindex or nosnippet tag holds the page back.
- Does this tool store the sites I check?
- No. It fetches the site's robots.txt, home page and llms.txt once, returns the result to your browser, and keeps nothing.