Free tool · AI crawler access

Can ChatGPT, Claude and Perplexity reach your site?

Enter a URL. We read your robots.txt the way each crawler does, request the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compare the answer with a normal browser request, and check that your content is in the raw HTML — these crawlers don't run JavaScript. You get a verdict for 31 crawlers in about 20 seconds. Background: can ChatGPT, Claude and Perplexity read your website?

Free · no login · results in about 20 seconds

What it checks

robots.txt, per crawler: Which of 31 search, AI-search, user-triggered and training crawlers your robots.txt allows on this URL, using RFC 9309 group matching — not a keyword search of the file.
Firewall and CDN blocks: The page requested with three AI-search user agents, compared with a browser request: 403s, HTTP 402 pay-per-crawl, and challenge pages. A robots.txt that says yes means nothing if the firewall says no.
Cloudflare settings: Whether the site is behind Cloudflare, what its managed robots.txt and Content-Signal lines say, and whether a training block could also catch search crawlers. See Cloudflare's 2026 AI crawl settings.
Content without JavaScript: How much of the page's text exists in the HTML the server returns. ChatGPT's, Claude's and Perplexity's crawlers read that HTML and nothing else.
Indexability and snippets: noindex, nosnippet, max-snippet and Bing's noarchive — the directives that keep a reachable page out of answers anyway.

The crawlers we check

Crawler	Operator	Job
OAI-SearchBot	OpenAI	AI search index
ChatGPT-User	OpenAI	Fetches when a user asks
GPTBot	OpenAI	Model training
OAI-AdsBot	OpenAI	Ads checks and site-owner services
Claude-SearchBot	Anthropic	AI search index
Claude-User	Anthropic	Fetches when a user asks
ClaudeBot	Anthropic	Model training
PerplexityBot	Perplexity	AI search index
Perplexity-User	Perplexity	Fetches when a user asks
Googlebot	Google	Search engine
Google-Extended	Google	Model training
Google-CloudVertexBot	Google	Ads checks and site-owner services
Google-Agent	Google	Browses and acts for a user
Google-GeminiNotebook	Google	Fetches when a user asks
Bingbot	Microsoft	Search engine
Applebot	Apple	Several uses, including training
Applebot-Extended	Apple	Model training
Meta-WebIndexer	Meta	Search engine
Meta-ExternalFetcher	Meta	Fetches when a user asks
Meta-ExternalAgent	Meta	Several uses, including training
Meta-ExternalAds	Meta	Ads checks and site-owner services
Amzn-SearchBot	Amazon	Search engine
Amzn-User	Amazon	Fetches when a user asks
Amazonbot	Amazon	Several uses, including training
DuckAssistBot	DuckDuckGo	Fetches when a user asks
MistralAI-Index	Mistral	AI search index
MistralAI-User	Mistral	Fetches when a user asks
MistralAI-Training	Mistral	Model training
YouBot	You.com	AI search index
Bytespider	ByteDance	Model training
CCBot	Common Crawl	Model training

Check it yourself

Three terminal commands reproduce the core of the test. Replace example.com with your domain.

1. Read your robots.txt

Look for a User-agent group naming the crawler, and for a catch-all User-agent: * group with Disallow: /.

curl -s https://example.com/robots.txt

2. Request the page as OAI-SearchBot and as a browser

Compare the status codes and sizes. A 403, a 402 or a much smaller body for the crawler means something at the edge treats it differently.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0" https://example.com/

3. Look for a firewall challenge

On Cloudflare, a cf-mitigated: challenge header means the request got a challenge page instead of your content.

curl -s -D - -o /dev/null -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/ | grep -i -E "^(HTTP|cf-mitigated|server)"

Questions

Is this the same as being cited by ChatGPT?+

No. Access is the first gate, not the last. A crawler that can't fetch your page can't cite it, but being reachable doesn't make a page cited — that depends on whether it answers the question and on what other sites say about you.

Why do you say the firewall test is simulated?+

We send requests with the crawlers' user-agent strings from our own servers. Real crawlers are verified by IP address and, increasingly, by signed requests, so a CDN may treat them differently from us in either direction. A block we see is a strong warning; a pass is good evidence, not proof.

Which crawlers matter most for being cited?+

The search-index crawlers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, plus Googlebot and Bingbot. Training crawlers such as GPTBot and ClaudeBot can be blocked without removing you from those answer engines.

How fast do robots.txt changes take effect?+

OpenAI and Perplexity both say about 24 hours for their search crawlers. Firewall rule changes apply immediately.

Does it cost anything?+

No. The audit is free, needs no account, and the report is kept for 24 hours.