Robots.txt Says Yes, the Firewall Says No: AI Search-Bot Access on the Top 5,000 Websites (October 2026)

On 13 October 2026 we checked the 5,000 most popular domains on the Tranco list. Among the 3,023 that served a home page, 11.6% disallow OpenAI’s training crawler GPTBot in robots.txt, but only 6.0% disallow OAI-SearchBot, the crawler behind ChatGPT search. The bigger gap is at the edge. On 9.8% of sites whose robots.txt allows OAI-SearchBot, PerplexityBot and Claude-SearchBot, a request identifying as at least one of them was refused, challenged or asked to pay, while a browser request got the page.

Not every edge block shuts out the real crawler. Our requests carry the crawler’s name but come from our own connection, not the vendor’s IP addresses. A second pass with two control requests suggests about 3 in 10 of these blocks target AI crawlers by name. The rest refuse any unknown bot or appear to check whether a claimed crawler is genuine. All figures below are from one run, from one network, on one day. The full per-site data is free to reuse: download the data (CSV).

Robots.txt says yes, the edge says no

robots.txt states a policy: crawlers read it and choose to comply. A CDN, web application firewall (WAF) or bot-management product enforces a policy, before robots.txt is ever consulted. When the two disagree, the firewall wins. We measured that disagreement only on sites where it means something. The browser request had to get a normal page, so sites that block every scripted client are left out. And robots.txt had to allow the crawler at “/”.

Taken together, 2,672 sites allow all three crawlers in robots.txt and answered all three requests. Of those, 263 (9.8%) turned at least one away and 166 (6.2%) turned away all three. Most refusals came as a challenge or block page: an HTTP 403 carrying bot-protection markup such as an “Access Denied” page or Cloudflare’s “Just a moment…” interstitial. Ten sites answered HTTP 402 Payment Required, the status Cloudflare’s pay per crawl returns to crawlers without a payment arrangement. The rate barely moves with popularity: 9.7% in the top 1,000 and 9.9% in ranks 1,001 to 5,000.

Some sites even name the crawler in robots.txt with an explicit Allow and still refuse it at the edge: 10 of the 173 sites that do this for OAI-SearchBot. That is the clearest sign that two teams, or two tools, set the policy and the enforcement separately.

Which of those blocks are actually aimed at AI?

A block on a request that claims to be OAI-SearchBot can mean three different things, and only one of them stops the real crawler for sure. So between 17:10 and 17:13 UTC, within an hour of the first pass, we requested every blocking site once more. We sent the browser and the three crawler user agents again, plus two controls. One was our own honest, unknown bot user agent (GEOAuditBot). The other was a spoofed Googlebot, which bot-management systems routinely refuse when it does not come from Google’s IPs. Of 667 site-and-crawler blocks, 658 (98.7%) repeated.

The AI-specific group, 77 sites or 2.9% of the 2,672, is the floor of real AI-search blocking we can see from outside. PerplexityBot was singled out most often: it was refused on 61 of the 77, against 37 for OAI-SearchBot and 17 for Claude-SearchBot. Among 298 comparison sites that had served all three crawlers, a spoofed Googlebot was refused on 7.1%, so verification of that kind is common even where AI crawlers are welcome.

Who blocks which crawler in robots.txt

We read each robots.txt the way a compliant crawler does under RFC 9309 and counted a bot as blocked when its governing group disallows the root path “/”. “Named explicitly” means a User-agent line names that bot. The rest are caught by a blanket “User-agent: *” with “Disallow: /”, which 3.1% of live sites have. Partial blocks, such as Disallow: /search, are not counted.

Three things stand out. First, the crawlers sites name are the old, well-known training crawlers: CCBot, Bytespider, GPTBot, ClaudeBot and Google-Extended. Crawlers documented more recently, such as MistralAI-Index, Google-GeminiNotebook and OAI-AdsBot, are almost never named; nearly all their “blocks” come from blanket Disallow-everything rules. Second, Googlebot and Bingbot are almost never named in a Disallow (0.2% and 0.3%). Third, the 287 sites that refused our scripted browser request are much stricter: 26.8% of them disallow GPTBot and 10.5% OAI-SearchBot. They include large publishers and marketplaces, and 186 of them are behind Cloudflare. Leaving them out understates blocking among the biggest sites, which is why the last column is there.

A note on user-triggered fetchers: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. A Disallow for those agents states a preference that may change nothing. Anthropic documents Claude-User as honouring robots.txt.

Training-only blockers: the split is real

OpenAI and Anthropic each run a training crawler and a separate search crawler, and OpenAI says the two settings are independent. The data shows many sites now use that split on purpose.

Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.

Across classes, 15.5% of live sites block at least one training crawler and 10.0% at least one AI search crawler. Of the training blockers, 36% allow all five AI search crawlers. The reverse almost never happens: 3 sites block an AI search crawler while allowing every training crawler. The Mistral row is the exception that proves the rule. Its crawlers are rarely named, so sites that block one usually block everything. If you want training opt-out without losing AI search, see GPTBot vs OAI-SearchBot for the exact lines.

By CDN: Cloudflare blocks AI crawlers by name, Akamai blocks unverified bots

We identified the CDN from home-page response headers, using the same check as our audit. Half the live sites send no header we recognise, so “none detected” includes CDNs that hide themselves and sites with no CDN at all.

Akamai-fronted sites turned away a simulated AI search crawler half the time, and almost never by name. Of the 97, 51 also refused our honest unknown bot and 42 served it but refused the spoofed Googlebot. In other words, they block bots in general or block bots that claim an identity they cannot prove. Most of these refusals were 403 “Access Denied” pages (85 of 97), usually with an AkamaiGHost server header. Whether the real, verified OAI-SearchBot gets through depends on how each site’s bot policy treats verified crawlers, which no outside test can see.

Cloudflare-fronted sites blocked less often, but when they did, the block was usually aimed at AI crawlers by name (31 of 42). That is consistent with Cloudflare’s own description of its free plan: “On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings.” Five of the ten HTTP 402 answers came from Cloudflare-fronted sites. Cloudflare changed its AI defaults on 15 September 2026; what each setting now does to Googlebot and the AI crawlers is in Is Cloudflare blocking Googlebot or ChatGPT?.

Content-Signal and Content-Usage: still rare

Content-Signal is Cloudflare’s robots.txt extension for stating how content may be used (search, ai-input, ai-train). Content-Usage is the IETF aipref working group’s draft equivalent (search, ai-use, train-ai). Neither binds a crawler. Adoption among the top sites is low.

Every site that publishes a Content-Signal line includes search=yes, and ai-input=yes outnumbers ai-input=no by 76 to 13. Seven files add Cloudflare’s experimental use=reference. Content-Signal is more common behind Cloudflare (42 of 638 robots.txt files on live Cloudflare sites, 6.6%) than elsewhere. Only 2 live sites sent a Content-Signal or Content-Usage HTTP response header on their home page. The near-absence of Cloudflare’s “BEGIN Cloudflare Managed content” block matches HasData’s September finding that it had disappeared from sites that carried it in summer. Our Content-Signal guide covers the syntax and who reads it.

No robots.txt, refusals and errors

• 79.9% of live sites publish a robots.txt file. Another 18.6% return a 404 or other 4xx (170), an HTML page at /robots.txt (239), an empty file (35) or a 401/403 to our crawler (117); RFC 9309 treats all of these as “allow everything”. The remaining 48 sites (1.6%) timed out, failed or answered with a 5xx or 429.
• Across all 5,000 domains, 54.9% served a robots.txt file. The list includes many hostnames with no website at the apex, such as CDN, DNS and telemetry domains: 902 did not resolve at all.
• 525 domains answered our browser request with HTTP 403 and are outside the “live sites” base. The 287 domains that refused it with a 4xx (mostly 403) but publish a robots.txt are discussed above.
• 3 live sites answered robots.txt with a 5xx or 429. RFC 9309 says a crawler must read a 5xx as “disallow everything”.
• Only 16 of 3,023 live sites challenged our browser request with a 200 interstitial; they are excluded from the edge figures.
• 1.7% of the 9,069 simulated crawler requests got no HTTP answer (timeouts and resets); those site-and-crawler pairs are excluded from edge rates.

How this compares with Ahrefs, Cloudflare Radar and HasData

Three earlier measurements are widely cited. Each answers a slightly different question, so the numbers are not directly comparable. The differences in method explain most of the gaps.

• Popularity drives blocking. Top sites name AI crawlers far more often than the long tail Ahrefs’ 140 million sites are dominated by: 9.3% of our live sites name and block GPTBot, against Ahrefs’ 0.50% explicit rate. Within our sample the top 1,000 block GPTBot more often (14.2%) than ranks 1,001 to 5,000 (11.0%).
• Denominators differ. Cloudflare counts robots.txt files, and its 37% applies to its own top-10,000 ranking, which includes many non-website hostnames. Our 54.9% across all domains and 79.9% across live sites are both higher. The two rankings and the 15 months between measurements both matter, so we would not read this as robots.txt adoption rising.
• Training bot vs search bot. HasData’s edge test sends GPTBot, a training crawler, from a datacenter IP. Datacenter traffic is treated more harshly than a fixed-line connection, and many sites deliberately refuse training crawlers. Our test sends the three AI search crawlers from a fixed-line connection, so our edge-block rate is not a substitute for theirs, and theirs is not a measure of AI search access.
• Matching rules. We read robots.txt as RFC 9309 specifies, so a blanket “User-agent: *” Disallow counts as a block. That matches Ahrefs’ “general blocks” and differs from HasData, which counts only a bot’s own group. Our “named explicitly” column is the comparable figure for HasData.
• The other direction. HasData found that 39.5% of sites disallowing GPTBot still served it a 200. For AI search crawlers we saw more: 73% of live sites whose robots.txt disallows OAI-SearchBot served our simulated OAI-SearchBot the page anyway. That is not hypocrisy. robots.txt is a request to crawlers, not something servers are expected to enforce.

What to do with this on your own site

1. Decide training and search separately. Blocking GPTBot, ClaudeBot or Google-Extended is a legitimate choice that does not affect AI search answers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those vendors’ answers.
2. Read your robots.txt per bot, not per file. Check which group each crawler actually obeys; a blanket “User-agent: *” Disallow catches every AI crawler you did not name.
3. Test the edge, not just the file. Request your home page with each crawler’s user agent and compare it with a browser request (commands below), or use the free AI crawler checker, which runs the same probes as this study.
4. If a crawler is refused, find out which layer did it. On Cloudflare, check AI Crawl Control and Security rules. On Akamai and other enterprise CDNs, check how bot-management policy treats verified and unverified crawlers. Then look in your server or CDN logs for requests from the vendor’s published IP ranges (openai.com/searchbot.json, perplexity.com/perplexitybot.json) and see what they received.
5. Treat Content-Signal as a statement, not a control. It is cheap to add, but crawlers are not bound by it and almost nobody publishes it yet.

# Compare a browser request with requests that identify as AI search crawlers.
# These are simulated: they come from your IP, not the vendor's, so a CDN may treat the real crawler differently.
URL='https://example.com/'
BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'
OAI='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
PPLX='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)'
CLAUDE='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/)'
for UA in "$BROWSER" "$OAI" "$PPLX" "$CLAUDE"; do
  curl -s -o /dev/null -w '%{http_code} %{size_download} bytes\n' -A "$UA" "$URL"
done

A 403, a 402 or a much smaller response for the crawler lines than for the browser line means an edge rule is involved. For the full set of gates a page must pass (robots.txt, CDN, raw HTML without JavaScript, snippet controls, response health), see Can ChatGPT, Claude and Perplexity read your website?. Or run a free audit, which reports each gate with its evidence.

Methodology

Sample

We used Tranco list 64VKX, generated on 12 October 2026. Tranco is a research ranking that combines five popularity sources (Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella) over 30 days, here 13 September to 12 October 2026, and keeps only registrable (pay-level) domains. We took ranks 1 to 5,000 exactly as listed, with no filtering. The list includes infrastructure domains that serve no website. They stay in the CSV, but rates are reported against “live sites”: the 3,023 domains whose home page returned a 2xx to a browser user agent. 2,976 of those also gave a readable robots.txt outcome, and that is the base for the robots.txt figures.

What we requested

1. https://domain/robots.txt with our own user agent, GEOAuditBot/1.0 (+https://geo-audit-saas-eight.vercel.app).
2. https://domain/ with a desktop Chrome user agent, following up to five redirects. If the host did not answer at all, we tried its www or bare sibling once.
3. If the home page landed on another origin (example.com to www.example.com), robots.txt again on that origin, because that is the file a crawler of that page obeys.
4. The final home-page URL three more times, identifying as OAI-SearchBot, PerplexityBot and Claude-SearchBot with the user-agent strings shown in the commands above. We sent these only if the browser request succeeded, as our audit does.

Each request had a 10-second timeout. robots.txt and the browser request were retried at most once, and only on a 5xx, a 429 or a network error. Crawler requests were never retried. At most 16 requests were in flight across all sites, and requests to any one site were strictly sequential. We fetched nothing beyond robots.txt and the home page. The crawl ran from 16:40 to 17:08 UTC on 13 October 2026, from a single fixed-line connection on an Indian ISP (not a datacenter or cloud IP). The Node.js HTTP client crashed natively several times. Each restart resumed after the last completed site and re-ran in-flight sites from scratch, so no site is counted twice. The second pass ran from 17:10 to 17:13 UTC.

All parsing and judging used our audit engine’s own code, unmodified. That covers the robots.txt parser and its 31-crawler registry, the fetcher, the edge-probe judge, CDN detection and the Content-Signal parser. The summary file documents every definition needed to reproduce each figure from the CSV.

How robots.txt was read

• Product tokens match exactly and case-insensitively, as RFC 9309 and Google’s reference parser do. “User-agent: Google” does not govern Googlebot, and Applebot-Extended does not govern Applebot.
• Groups that name the same token are combined, the longest matching rule wins, and Allow wins a tie. A crawler with no group of its own obeys “User-agent: *”. Applebot follows Googlebot’s group when it is not named, as Apple documents.
• “Blocked” means the governing group disallows “/”. A 4xx, an HTML page at /robots.txt or an empty file means everything is allowed (RFC 9309 §2.3.1.3). A 5xx means complete disallow (§2.3.1.4). We saw only 3 of those on live sites and kept them out of the block rates. Sites whose robots.txt timed out are excluded.

What “blocked at the edge” means

• Refused: any non-2xx answer without challenge markup (mostly 403; also 406, 410, 429, 5xx).
• Challenge: a non-2xx answer, or a near-empty 2xx, that carries bot-protection markup. Examples are a cf-mitigated: challenge header, “Just a moment…”, “Access Denied”, captcha-delivery, px-captcha, Incapsula or Sucuri pages. A 2xx counts only if it has fewer than 40 words, or under 30% of the words the browser received.
• Paywall: HTTP 402 Payment Required.
• Only sites where the browser request got a normal page count, and only crawlers that robots.txt allows at “/”.

Limitations

• Simulated user agent, not IP-verified. The crawler requests came from our connection. Sites that verify crawler IPs may serve the real crawler, or block it more. The second-pass controls narrow this down but cannot settle it for any single site.
• One vantage point, one run, one day. Sites geo-block, rate-limit and change rules. A datacenter IP, another country or another week could give different numbers. The 98.7% repeat rate covers minutes, not weeks.
• Home page only. A site can allow the home page and block /blog/ or /products/ in robots.txt or at the edge; partial blocks are not counted.
• Top 5,000 only. Popular sites lean towards large companies on enterprise CDNs, so these rates do not describe small business websites.
• Our “browser” is a plain HTTP client with a Chrome user agent, not a real browser. 525 domains refused it with a 403 and are outside the edge figures, which likely understates edge blocking among the most defended sites.
• We fetched robots.txt as GEOAuditBot. A site that refuses unknown bots may have served a real crawler a different file; 117 live sites answered 401 or 403, counted as “no rules” per RFC 9309.
• CDN detection relies on response headers, so “none detected” mixes self-hosted sites with CDNs that send no identifying header.
• We describe what sites served, not why. A CDN being present does not mean the CDN’s default caused a block; the site owner may have written the rule.

Download the data

Download the data (CSV) has one row per domain, 5,000 rows and 62 columns. It holds robots.txt status and size, the verdict at “/” for each of 31 crawlers, wildcard Disallow, Content-Signal and Content-Usage values, the detected CDN, browser status, size and word count, and status, size and verdict for each simulated crawler request. A second-pass class sits on each edge-blocking site. The summary file (JSON) documents every column, definition and aggregate in this post. It contains no personal data, only domains and the responses their servers gave.

Both files are licensed CC BY 4.0. Please credit “SEO·AEO·GEO, AI crawler access study, October 2026” and link to this page. The checks themselves are documented, with their evidence grades, on our methodology page.