Mayexa Insights

AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and Google-Extended

AI crawlers read websites for AI training, AI search and user requests. See what GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended do, a robots.txt that allows AI search but blocks training, and how to check your site.

4 min read
Visual guide to know your ai crawler rules: crawler identity, robots.txt, review permissions. Mayexa Technologies.

Part of the AEO and GEO guide. Checked against OpenAI, Anthropic, Perplexity and Google documentation on 20 October 2026. Crawler names change; confirm in each company's documentation before relying on them.

What are AI crawlers?

AI crawlers are bots that AI companies use to read web pages. Some collect pages for training AI models, some build the index behind AI search answers and citations, and some fetch a page only when a user asks a question about it. Each has its own name, called a user agent, which you can allow or block in robots.txt.

The key decision for a business is simple: allow the search and user bots if you want to appear in AI answers, and decide separately whether you want your content used for training.

Which AI crawlers should you know?

The main AI companies publish separate bots for search and for training, so you can choose. The table lists the most important ones, what each does and whether it follows robots.txt, taken from each company's own documentation. Blocking a search bot can remove you from that engine's answers; blocking a training bot does not.

User agentCompanyUsed forrobots.txt
OAI-SearchBotOpenAIChatGPT search results and citationsFollowed; blocked sites are not shown in ChatGPT search answers
ChatGPT-UserOpenAIPages ChatGPT opens for a user's requestMay not apply to user-initiated actions
GPTBotOpenAITraining OpenAI modelsFollowed; blocking stops training use, not search
Claude-SearchBotAnthropicClaude search result qualityFollowed
Claude-UserAnthropicPages Claude fetches for a user's questionFollowed
ClaudeBotAnthropicTraining Anthropic modelsFollowed, including Crawl-delay
PerplexityBotPerplexityPerplexity search results; not used for trainingManaged through robots.txt
Perplexity-UserPerplexityPages fetched for a user's questionGenerally ignores robots.txt
Google-ExtendedGoogleToken for Gemini training and grounding, not a separate crawlerDoes not affect Google Search inclusion or ranking

How do you allow AI search but block AI training?

Add separate robots.txt groups: allow the search bots and block the training bots by name. The example below lets ChatGPT, Claude and Perplexity search read your site while opting out of OpenAI and Anthropic model training and Google's Gemini training. Keep your normal rules for all other crawlers below it.

# AI search: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Whether to block training is a business choice. Allowing it may help models know about your brand in the long run; blocking it keeps your content out of future training sets. Blocking training does not remove you from AI search answers.

What happens if you block all AI bots?

If you block the search bots, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, those engines cannot read your pages for search, so you are much less likely to be cited or linked in their answers. OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, apart from navigational links.

A common mistake is a broad rule such as User-agent: * with Disallow: /, often left over from a staging site, which blocks every crawler including Google.

How do you check which AI crawlers your site allows?

Open yoursite.com/robots.txt and look for the user agents above, or use a checker. Mayexa's free AI Visibility audit reads your robots.txt and shows, for each major AI crawler, whether your homepage and sample pages are allowed, and flags search bots that are blocked by mistake.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, so block or allow that one to control appearance in ChatGPT search answers.

Does Google-Extended affect AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini training and grounding. AI Overviews are part of Google Search and use Googlebot.

How quickly do robots.txt changes take effect?

It depends on each crawler. OpenAI notes that changes for its search bot can take about 24 hours to be reflected.

Sources

Who writes Mayexa Technologies pages?

The Mayexa Editorial Team writes and updates the service, course, product, tool and guide pages on mayexa.in. Each page shows the date it was last updated. To report an error or ask a question about a page, use the contact page.