How to Allow AI Crawlers in robots.txt: GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-SearchBot
Which AI crawlers affect your visibility in ChatGPT, Claude, Perplexity and Google, which only collect training data, and the exact robots.txt rules to use.
Updated · 3 min read
On this page
“Should I block AI bots?” is the wrong question, because there isn’t one AI bot. OpenAI, Anthropic, Perplexity and Google each run several crawlers with different purposes. Some decide whether you can appear in AI answers. Others only collect data for training future models. Treat them the same and you either give away more than you meant to, or vanish from AI search without noticing.
The crawlers that matter, by purpose
The descriptions below come from each company’s own documentation.
| Crawler | Company | What it does | Block it and… |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT’s search features | You can drop out of ChatGPT search results |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI’s models | Your content is excluded from training; search is unaffected |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks; not automatic crawling | OpenAI says robots.txt rules may not apply |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude’s search answers | Anthropic says it may reduce your visibility in user search results |
| ClaudeBot | Anthropic | Collects content that could contribute to training | Future content is excluded from training |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | May reduce visibility for user-directed web search |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; not used for training | You can drop out of Perplexity results |
| Perplexity-User | Perplexity | Visits pages for user questions | Perplexity says it generally ignores robots.txt |
| Googlebot | Google Search, including AI Overviews and AI Mode | You leave Google Search entirely | |
| Google-Extended | A control token for Gemini training and grounding in the Gemini app | Gemini can’t use or cite you; Search and AI Overviews unaffected | |
| Bingbot | Microsoft | Bing search, which also powers Microsoft Copilot answers | You leave Bing and Copilot answers |
Step 1: Check what you block today
Open yourdomain.com/robots.txt in a browser, or run your domain through our free AI Crawler Checker, which reads your live file and groups every AI crawler by what blocking it does.
Look for groups like this:
User-agent: OAI-SearchBot
Disallow: /A Disallow: / under a search crawler means that engine can’t index your site.
Step 2: Decide your policy
Most businesses that want to be found in AI answers but would rather not contribute to model training settle on this:
- Allow all search crawlers and user-request fetchers.
- Allow or block Google-Extended depending on whether you want to be cited in the Gemini app.
- Your choice on training crawlers (GPTBot, ClaudeBot, Applebot-Extended, CCBot).
Step 3: Use rules like these
# AI search and answer engines: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
# Training-only crawlers: optional. Remove these two groups if you're happy to be used for training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlA crawler follows the most specific group that names it, so a User-agent: * group with Disallow: / blocks every crawler that doesn’t have its own group. If you have one of those, make sure each search crawler has its own Allow: / group.
Step 4: Edit robots.txt in WordPress
- Yoast SEO: Yoast SEO → Tools → File editor.
- Rank Math: Rank Math → General Settings → Edit robots.txt.
- No plugin: upload a
robots.txtfile to your site’s root folder (on Hostinger: File Manager → public_html).
Also check your security plugin and CDN (for example Cloudflare’s bot settings). Those can block AI crawlers at the server level even when robots.txt allows them.
Step 5: Re-test
Run the AI Crawler Checker again to confirm the live file changed. Crawler access is only the first gate. Once it’s open, the AI Visibility Fixer checks what comes next: entity data, FAQs, local signals and profiles.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
According to OpenAI, GPTBot is used for training data. Search visibility in ChatGPT is controlled by OAI-SearchBot. Blocking GPTBot alone does not opt you out of ChatGPT search.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls Gemini training and grounding in the Gemini app. AI Overviews and AI Mode are part of Google Search and follow Googlebot and the normal Search controls.
What if I have no robots.txt at all?
Then every crawler is allowed by default. That is fine for AI visibility, though most sites benefit from a robots.txt that lists their sitemap.
The Onxeera team builds tools that help businesses get found, understood and recommended by AI search engines like ChatGPT, Perplexity, Gemini and Google AI Overviews.
Can AI engines read your website?
Check your robots.txt against every major AI crawler in seconds. Free, no sign-up.