Free tool · AI robots.txt
AI robots.txt generator: stay in AI search, opt out of training
Every AI company now runs separate crawlers for search and for training, so one robots.txt can keep you in ChatGPT, Claude and Perplexity answers while opting out of model training. Pick a policy below and copy the result. It's built from the same 31-crawler registry our audit uses. Then run the free audit below it to check that your CDN or firewall agrees. Background: GPTBot vs OAI-SearchBot
Generate your robots.txt
# robots.txt: Stay in AI search, opt out of training (recommended)
# Generated with https://seoaeogeo.app/tools/ai-robots-txt-generator
# 7 crawlers that collect training data
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot, Bingbot and the AI search crawlers
User-agent: *
Allow: /
Merge these lines into your existing robots.txt rather than replacing rules you already rely on. User-triggered fetchers (ChatGPT-User, Perplexity-User, Google-Agent, Google-GeminiNotebook, Meta-ExternalFetcher, Amzn-User) may not follow robots.txt because a person asked for the page; their vendors say so. A CDN or firewall can still block crawlers this file allows, so check the result with the audit below.
Free · no login · results in about 20 seconds
What it checks
- Training opt-out without losing AI search
- Blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, MistralAI-Training, CCBot and Bytespider; leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot and the search engines allowed.
- Mixed crawlers handled honestly
- Applebot, Amazonbot and Meta-ExternalAgent do search and training under one name, so the recommended policy leaves them allowed and says why.
- Valid, grouped rules
- One group of User-agent lines with a single Disallow, per RFC 9309, plus an optional Sitemap line.
- What robots.txt can't do
- User-triggered fetchers may ignore it, and a firewall can block crawlers it allows. The audit above checks both.
Every AI crawler, by job (31)
| robots.txt token | Operator | Job | If you block it |
|---|---|---|---|
| Claude-SearchBot | Anthropic | AI search index | Blocking removes you from that assistant's search answers and citations |
| MistralAI-Index | Mistral | AI search index | Blocking removes you from that assistant's search answers and citations |
| OAI-SearchBot | OpenAI | AI search index | Blocking removes you from that assistant's search answers and citations |
| PerplexityBot | Perplexity | AI search index | Blocking removes you from that assistant's search answers and citations |
| YouBot | You.com | AI search index | Blocking removes you from that assistant's search answers and citations |
| Amzn-User | Amazon | Fetches when a user asks | May ignore robots.txt (user-triggered) |
| Claude-User | Anthropic | Fetches when a user asks | May ignore robots.txt; blocking stops on-demand reads where honoured |
| DuckAssistBot | DuckDuckGo | Fetches when a user asks | May ignore robots.txt; blocking stops on-demand reads where honoured |
| Google-GeminiNotebook | Fetches when a user asks | May ignore robots.txt (user-triggered) | |
| Meta-ExternalFetcher | Meta | Fetches when a user asks | May ignore robots.txt (user-triggered) |
| MistralAI-User | Mistral | Fetches when a user asks | May ignore robots.txt; blocking stops on-demand reads where honoured |
| ChatGPT-User | OpenAI | Fetches when a user asks | May ignore robots.txt (user-triggered) |
| Perplexity-User | Perplexity | Fetches when a user asks | May ignore robots.txt (user-triggered) |
| Amzn-SearchBot | Amazon | Search index | Blocking removes you from that search engine, and from AI features built on it |
| Googlebot | Search index | Blocking removes you from that search engine, and from AI features built on it | |
| Meta-WebIndexer | Meta | Search index | Blocking removes you from that search engine, and from AI features built on it |
| Bingbot | Microsoft | Search index | Blocking removes you from that search engine, and from AI features built on it |
| Amazonbot | Amazon | Search and training | One user agent for both jobs; blocking also removes the search use |
| Applebot | Apple | Search and training | One user agent for both jobs; blocking also removes the search use |
| Meta-ExternalAgent | Meta | Search and training | One user agent for both jobs; blocking also removes the search use |
| ClaudeBot | Anthropic | Model training | Opt out of training; no effect on AI search answers |
| Applebot-Extended | Apple | Model training | Opt out of training; no effect on AI search answers |
| Bytespider | ByteDance | Model training | Opt out of training; no effect on AI search answers |
| CCBot | Common Crawl | Model training | Opt out of training; no effect on AI search answers |
| Google-Extended | Model training | Opt out of training; no effect on AI search answers | |
| MistralAI-Training | Mistral | Model training | Opt out of training; no effect on AI search answers |
| GPTBot | OpenAI | Model training | Opt out of training; no effect on AI search answers |
| Google-Agent | Acts for a user | May ignore robots.txt (user-triggered) | |
| Google-CloudVertexBot | Ads checks and owner services | Unrelated to AI answers | |
| Meta-ExternalAds | Meta | Ads checks and owner services | Unrelated to AI answers |
| OAI-AdsBot | OpenAI | Ads checks and owner services | Unrelated to AI answers |
Free to run. Pay only for the fixes you want.
Free
No login
Your score, every check's result, and the full fix for the top 3 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.
Questions
Will blocking GPTBot remove me from ChatGPT?+
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, which you can keep allowed. OpenAI documents each setting as independent.
Does blocking Google-Extended affect AI Overviews?+
No. Google-Extended controls Gemini training and grounding outside Search. AI Overviews and AI Mode use Googlebot, so blocking Google-Extended doesn't remove you from them.
How long until changes take effect?+
OpenAI and Perplexity say about 24 hours for their crawlers to pick up a robots.txt change. CDN and firewall rules apply immediately.
Should I block every AI crawler?+
Only if you'd rather not appear in AI assistants at all. Blocking the AI search crawlers removes you from their answers and citations, which for most businesses means losing a growing source of referrals.
Is this robots.txt enough on Cloudflare?+
Check Cloudflare's AI Crawl Control too: its Block setting can stop crawlers your robots.txt allows. See our guide to Cloudflare's 2026 settings.