Is your robots.txt blocking AI search engines?

The five names that matter

These are the crawlers a site check tests against; if any is disallowed from “/”, that engine cannot fetch the site to cite it. Check robots.txt for a line naming one of these under a “Disallow: /” or “Disallow: /*” rule:

• OAI-SearchBot — fetches pages for ChatGPT's live web search citations.
• ChatGPT-User — fetches a page when a user asks ChatGPT to browse or read it directly.
• PerplexityBot — fetches pages for Perplexity's answers.
• Google-Extended — controls use for Gemini and Google's AI features (AI Overviews, AI Mode), separate from ordinary Search.
• ClaudeBot — fetches pages for Claude's web search citations.

Two names that are often confused for these

GPTBot is OpenAI's training crawler — blocking it opts a site out of being used to train future models, but it has nothing to do with whether ChatGPT can cite the page in an answer today; that's OAI-SearchBot. Googlebot is classic Google Search — blocking it removes a site from Google Search entirely, a far bigger and usually unintended step, and is a separate decision from Google-Extended.

A site check reports these two for context but doesn't treat blocking them as an AI-visibility problem in the same way: GPTBot only affects training, and Googlebot blocking is a much larger, deliberate call most sites should never make by accident.

What to add

Most sites that block these do it by accident, usually with a blanket “Disallow: /” under “User-agent: *” meant for something else, or a security plugin's default block list. To explicitly allow the five crawlers above regardless of a wildcard rule, add a group for each above the wildcard block:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

Check your own

A free scan reads a site's actual robots.txt and reports exactly which of the five crawlers, if any, are blocked and which rule causes it — no need to read the file by hand.