Reference · updated 18 October 2026
The 31 crawlers and robots.txt tokens our audit checks, grouped by company, with each operator's own user-agent string (24 of them publish one), what the crawler is for, whether it follows robots.txt, and where to verify its IP addresses. Search crawlers decide whether AI answers can cite you; training crawlers only decide what goes into future models. To act on it, use the AI robots.txt generator, then run the audit below to see what your server and CDN actually let through.
Free · no login · results in about 20 seconds
- robots.txt, per crawler
- Reads your robots.txt the way RFC 9309 says each of these 31 crawlers must, so a group for GPTBot never gets mistaken for one about OAI-SearchBot.
- What your CDN lets through
- Requests the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compares the answers with a browser request, which catches firewall blocks robots.txt can't show.
- Content without JavaScript
- AI crawlers don't run JavaScript. The audit checks how much of the page is in the raw HTML they receive.
- Snippet controls
- nosnippet and max-snippet keep pages out of AI Overviews while they still rank. The audit flags them.
Free
No login
Your score, every check's result, and the full fix for the top 3 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.
What is ChatGPT's user agent?+
ChatGPT uses three. OAI-SearchBot builds the index ChatGPT search cites (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot). ChatGPT-User fetches a page when someone asks ChatGPT to read it. GPTBot collects training data. Blocking GPTBot doesn't remove you from ChatGPT search.
Which AI crawlers may ignore robots.txt?+
Per their operators: ChatGPT-User, Perplexity-User, Google-Agent, Google-GeminiNotebook, Meta-ExternalFetcher, Amzn-User. Most are user-triggered fetchers, where a person asked for the page, so the operators treat robots.txt as not applying.
How do I check that a crawler is genuine?+
Anyone can copy a user-agent string, so check the IP address. Operators publish IP ranges for OAI-SearchBot, ChatGPT-User, GPTBot, OAI-AdsBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-CloudVertexBot, Google-Agent, Bingbot, Applebot, Amzn-SearchBot, Amzn-User, Amazonbot, MistralAI-Index, MistralAI-User, DuckAssistBot, CCBot; the links are under each crawler. Google, Bing and Apple also support a reverse DNS lookup on the IP, followed by a forward lookup to confirm it.
Does blocking Google-Extended affect Google Search or AI Overviews?+
No. Google-Extended is a robots.txt token, not a crawler. It controls Gemini training and grounding outside Search. AI Overviews and AI Mode use Googlebot.
How often is this list updated?+
When an operator adds, renames or retires a crawler, and at least every quarter. The date at the top is when we last checked every vendor page.