Can ChatGPT, Claude and Perplexity Read Your Website? A 5-Gate Test You Can Run Yourself

Usually yes, but you can't assume it. For ChatGPT, Claude or Perplexity to read and cite a page, that company's search crawler has to pass five gates: robots.txt lets it in, your CDN or firewall doesn't turn it away, the words are in the raw HTML (these crawlers don't run JavaScript), nothing on the page says “don't index” or “don't quote”, and the server answers cleanly. You can test all five with curl in about ten minutes, or with our free AI crawler checker. One caveat matters more than any other: a test from your own computer is a simulation, because CDNs recognise real crawlers by their IP addresses and signatures, not by the name they send.

This guide gives the exact commands, shows what they returned on five well-known sites on 13 October 2026, and explains how to read the results without fooling yourself. The commands run in a macOS or Linux terminal; on Windows use WSL or Git Bash, which include curl, grep and perl.

Which crawler reads your site for each assistant

Each assistant runs separate bots for separate jobs. Only the search bot decides whether you can be found and cited later. The user-triggered fetcher visits a page when someone asks about it right now. The training bot has nothing to do with citations. Blocking the wrong one is the most common mistake we see, and the GPTBot vs OAI-SearchBot guide covers OpenAI's split in detail.

Meta's and Amazon's newer bots work differently from what most guides say; see Meta-WebIndexer and Amzn-SearchBot explained. For Google, AI Overviews and AI Mode use the normal Search index, so the Googlebot rules you already have apply.

The five gates at a glance

Gate 1: robots.txt names the right bot

Start with the file itself, and with its status code. Under RFC 9309, a robots.txt that returns a 4xx means a crawler “MAY access any resources on the server”, while a 5xx means the crawler “MUST assume complete disallow”. A robots.txt that errors is worse than none.

# Read the whole file (it is usually short)
curl -s https://www.example.com/robots.txt

# Its status code: 200 is normal, 4xx means "no rules", 5xx means "stay out"
curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/robots.txt

# The groups that matter for AI search (stacked user-agent lines share the rules below them)
curl -s https://www.example.com/robots.txt | grep -i -A6 -E '^user-agent: *(\*|oai-searchbot|chatgpt-user|claude-searchbot|claude-user|perplexitybot|googlebot|bingbot|meta-webindexer|amzn-searchbot)'

Three rules decide the verdict. First, matching is by exact token. RFC 9309 requires “case-insensitive matching to find the group that matches the product token”, and Google's reference parser compares the token with a case-insensitive equality check, not a prefix. So User-agent: Claude does not govern ClaudeBot, and a group for GPTBot says nothing about OAI-SearchBot. Second, a crawler obeys the group that names it and ignores the * group; if it isn't named, the * group applies. Third, changes are not instant: OpenAI says “it can take ~24 hours from a site's robots.txt update for our systems to adjust”, Perplexity says “up to 24 hours”, and Meta says its crawlers “may cache the contents of robots.txt for up to 24 hours”.

What we saw on 13 October 2026: reddit.com serves User-agent: * with Disallow: / and names no bot at all, so OAI-SearchBot, Claude-SearchBot and PerplexityBot are all disallowed by the file. theguardian.com doesn't name OAI-SearchBot (so its * group applies, which only blocks some paths) but gives PerplexityBot, ClaudeBot, Claude-SearchBot and Claude-User Disallow: / groups. medium.com disallows GPTBot, ClaudeBot and other training bots and leaves OAI-SearchBot under its * group. stackoverflow.com answered our robots.txt request with HTTP 418 and a deny-all file whatever user agent we sent. That last one is a warning: the robots.txt your laptop receives is not necessarily the one a verified crawler receives.

Gate 2: the CDN or firewall lets the request through

This is the gate robots.txt checkers miss. A bot-management rule, an “AI bots” toggle or a pay-per-crawl setting refuses the request before your server sees it, and nothing in your HTML or robots.txt shows it. The test: request the same URL as a browser and as each AI crawler, then compare the status code and the size of what came back.

URL='https://www.example.com/'
BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
OAI='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
PPLX='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)'
# Anthropic does not publish full user-agent strings; this one carries the token rules match on
CLAUDE='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/)'

for UA in "$BROWSER" "$OAI" "$PPLX" "$CLAUDE"; do
  curl -s --compressed -o /dev/null -A "$UA" -w '%{http_code}  %{size_download} bytes\n' "$URL"
done

# Then look at the headers a crawler request gets back
curl -s --compressed -D - -o /dev/null -A "$OAI" "$URL" | grep -i -E '^(HTTP|server|cf-ray|cf-mitigated|x-amzn-waf-action)'

The OAI-SearchBot and PerplexityBot strings are copied from OpenAI's and Perplexity's documentation. Cloudflare marks challenge responses with a cf-mitigated header (“challenge is the only valid value”), and in our tests an AWS WAF challenge came back as a 202 with an empty body and an x-amzn-waf-action: challenge header. An HTTP 402 means Cloudflare pay per crawl, where crawlers “receive an HTTP 402 Payment Required response with pricing”. Here is what the loop returned on 13 October 2026 between 16:24 and 16:31 UTC, from one machine using curl 8.19:

How to read your own results: if the crawler gets the same status and a similar size as the browser, the gate is probably open. If the browser gets a 200 and the crawler gets a 403, 402, a challenge or a much smaller page, a rule is very likely targeting that crawler. If the browser request fails too, curl itself is being treated as a bot and the test tells you nothing; that happened on two of our five sites. On Cloudflare, the place to look next is AI Crawl Control and Security → Bots; our Cloudflare guide covers what changed on 1 July and 15 September 2026.

Why a user-agent test is only a simulation

Changing the user agent changes one header. It does not change where the request comes from. CDNs decide who is a real crawler by other means. Cloudflare's definition of a verified bot is that it “declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS”. So your spoofed request and the real crawler can get opposite answers: a CDN may refuse your fake OAI-SearchBot as an impostor while letting the real one through, or allow your fake while a rule aimed at verified AI crawlers turns the real one away.

The proof is in your logs. Find requests that claim to be the bot, then check their source IPs against the operator's published list. OpenAI, Perplexity and Anthropic all publish JSON lists; Google documents a reverse-then-forward DNS check for its crawlers.

# Who claims to be OAI-SearchBot? (combined log format; change the path for your server)
grep 'OAI-SearchBot' /var/log/nginx/access.log | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head

# The published ranges to compare against
curl -s https://openai.com/searchbot.json
curl -s https://www.perplexity.com/perplexitybot.json
curl -s https://claude.com/crawling/bots.json

# Google: reverse DNS must end in googlebot.com or google.com, then resolve back to the same IP
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The second column in that log summary is the status code your server returned. Hundreds of hits from the published ranges with status 200 is a pass. No hits at all, or hits that all ended in 403, is the finding. If you only have CDN logs, filter by the verified-bot field your CDN provides rather than by user agent.

Gate 3: the words are in the raw HTML

Vercel and MERJ analysed crawler traffic and found that “none of the major AI crawlers currently render JavaScript”; ChatGPT's and Claude's crawlers fetch JavaScript files but “don't execute them”. Googlebot renders pages, and Vercel reports that Applebot does too, but OAI-SearchBot, Claude-SearchBot and PerplexityBot should be treated as reading only the HTML your server sends. If your product names, prices or answers are injected by JavaScript, those crawlers see a page without them.

# Copy a sentence you can see on the page in a browser. Is it in the HTML a crawler gets?
curl -s --compressed -A "$OAI" "$URL" | grep -c 'a sentence copied from your page'

# Roughly how many words of visible text that HTML contains (scripts and styles removed)
curl -s --compressed -A "$OAI" "$URL" | perl -0777 -pe 's/<script\b.*?<\/script>//gis; s/<style\b.*?<\/style>//gis; s/<[^>]+>/ /g' | wc -w

A count of 0 for your sentence means crawlers that don't render can't quote it. On 13 October 2026 the word count, fetched with the OAI-SearchBot string, was 1,236 for cloudflare.com, 2,726 for bbc.com and 3,213 for theguardian.com/international. For reddit.com it was 1. Reddit answered our browser and crawler strings alike with a 200 and the same 8,397-byte page, whose only visible word was its title and whose script solves a challenge and resubmits before anything loads. A status code check would have called that a pass. Always look at what came back, not just the number. Clean, semantic markup helps once the text is there; see semantic HTML for AI crawlers.

Gate 4: nothing tells engines not to index or quote

A page can be reachable and readable and still be excluded by its own instructions. noindex removes it from search indexes. For Google, nosnippet “will also prevent the content from being used as a direct input for AI Overviews and AI Mode”, and max-snippet limits how much can be used. For Bing and Copilot, “Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers.” These controls can sit in a meta tag or in an X-Robots-Tag HTTP header, which is easy to forget because it never appears in the page source.

# Header-level controls (often set by a CDN or framework, invisible in the HTML)
curl -sI -A "$OAI" "$URL" | grep -i '^x-robots-tag'

# Meta robots tags, including bot-specific ones such as googlebot or bingbot
curl -s --compressed -A "$OAI" "$URL" | grep -i -o '<meta[^>]*robots[^>]*>'

# Sections hidden from snippets
curl -s --compressed -A "$OAI" "$URL" | grep -c 'data-nosnippet'

On 13 October 2026 the bbc.com home page sent X-Robots-Tag: bingbot: noarchive. Under Bing's published rules that keeps the page out of Copilot answers while leaving it in Bing search results. That is a deliberate publisher choice, but the same header set by accident on a business site would quietly remove it from Copilot. One Google control can't be tested from outside at all: Google now also requires a site to be included in generative AI features in Search Console (the include/exclude control has been available worldwide since 31 August 2026), and only the site owner can see that setting. The AI Overviews eligibility guide covers each control and the Google-Extended myth.

Gate 5: the server answers cleanly and quickly

The last gate is ordinary web health, seen from a crawler's side: a final 200, few redirects, a reasonable time to first byte, and a page that isn't an error dressed up as a 200 (a soft 404). No AI operator publishes its timeout, so we won't quote one, but every extra hop and every slow response is another chance for a fetch to be abandoned.

# Final status, number of redirects, time to first byte, size and final URL
curl -s -o /dev/null -L -A "$OAI" -w 'status %{http_code}, %{num_redirects} redirects, first byte %{time_starttransfer}s, total %{time_total}s, %{size_download} bytes, %{url_effective}\n' 'http://example.com/'

# Every hop in the chain
curl -sIL -A "$OAI" 'http://example.com/' | grep -i -E '^(HTTP|location)'

Example from 13 October 2026: http://theguardian.com/ reached its final page through three redirects (301 to https, 301 to www, 302 to /international) with a first byte after 0.50 seconds. That is healthy, but on a smaller server the same chain plus a slow origin adds up. Link to your final URLs directly, and keep sitemaps free of redirecting addresses. More on timeouts and budgets in fetch budgets for AI crawlers.

Run all five gates in one free check

If you would rather not run the commands, the free AI crawler checker does the same work on any URL in about 20 seconds. It reads your robots.txt the way each crawler does, with RFC 9309 group matching, and gives a verdict per bot for the search, AI-search, user-triggered and training crawlers. It requests the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compares each response with a browser fetch, flagging 403s, 402 pay-per-crawl responses and challenge pages. It measures how much of your text is in the raw HTML, and checks noindex, nosnippet, max-snippet and Bing's noarchive. Every edge result is labelled as simulated, for the reasons above. It deliberately does not send fake Googlebot or Bingbot requests: CDNs routinely block requests that claim to be Googlebot from outside Google's IP ranges, so a spoofed test would report a block on almost every protected site.

For the wider picture (structure, freshness, entity signals), run a free audit. The methodology lists every check with its evidence grade.

What to fix first

1. Fix any 5xx on robots.txt today. Crawlers that follow RFC 9309 treat it as “complete disallow”.
2. Make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot are not disallowed, either by name or through a User-agent: * group with Disallow: /. Decide training bots (GPTBot, ClaudeBot) as a separate policy.
3. Where a crawler string gets a 403, 402 or challenge while a browser gets the page, find the rule in your CDN or WAF and allow the verified search crawlers. Then confirm in logs that requests from the published IP ranges get a 200.
4. Move key text (what you do, prices, answers, contact details) into server-rendered HTML.
5. Remove accidental noindex, nosnippet and noarchive, including in X-Robots-Tag headers, and check the Search Console generative AI setting if you own the site.
6. Collapse redirect chains and link to final URLs.

Passing all five gates doesn't make an assistant recommend you. It makes you eligible. Whether you are chosen depends mostly on what others say about you across the web, which no crawler test measures. But if a gate is shut, nothing else you do on the page can be read.