Robots.txt Allows AI Crawlers, the Firewall Blocks Them: 131 Top Sites Where the Two Disagree

A site can say yes to AI search crawlers in robots.txt and still turn them away. The file is one decision, made in a text file. The firewall is another, made by a CDN, WAF or bot manager on every request. When the two disagree, the site owner often doesn’t know, because nothing in robots.txt tells you what the edge does.

We went back to the data from our October 2026 crawler study and pulled out every listed site where robots.txt allows all three AI search crawlers (OAI-SearchBot for ChatGPT search, PerplexityBot, and Claude-SearchBot) but a request carrying one of those names was refused, challenged or asked to pay while an ordinary browser request got the page.

robots.txt and the firewall are two separate decisions

RFC 9309, the Robots Exclusion Protocol, is explicit that its rules “are not a form of access authorization”. A crawler fetches robots.txt and chooses to follow it. Nothing in the file is enforced. Enforcement, if any, happens at the edge: the CDN or firewall in front of the site looks at each request and decides whether to serve it, challenge it, or refuse it.

Those edge systems increasingly treat “claims to be a known bot” as something to check. Cloudflare’s verified-bots documentation describes bots identifying themselves through a published IP list with a stable user-agent, reverse DNS, or Web Bot Auth signatures. Akamai says its Bot Manager keeps a directory of known bots. OpenAI publishes the IP addresses OAI-SearchBot uses (openai.com/searchbot.json), and Perplexity does the same for PerplexityBot (perplexity.com/perplexitybot.json). A request that carries one of those names from somewhere else looks like an impostor, and blocking impostors is a reasonable thing for a firewall to do.

RFC 9309 also says what a crawler does when robots.txt itself can’t be read. A 4xx answer means the crawler may access anything; a 5xx or network failure means it must assume complete disallow. That matters for the wider count below.

How we counted

• Data: the per-site results of our 13 October 2026 study of the Tranco top 5,000, as shipped on our /ai-visibility pages. “Listed” means every site with a public page there except adult sites, which we exclude as the study does: 3,146 sites.
• Readable robots.txt: we read rules for all three AI search crawlers on 2,543 of them. The rest had no file, an empty file, HTML instead of text, a refusal, or a timeout.
• Allows all three: none of OAI-SearchBot, PerplexityBot or Claude-SearchBot is disallowed at the root, by name or by the * group: 2,229 sites. Of these, 2,023 also answered our plain browser request cleanly, which is needed before a crawler-named block can be told apart from blocking all automation.
• Blocked at the edge: the browser request got the page, and our request with the crawler’s User-Agent got a refusal (a 4xx or 5xx status), a bot challenge page, or HTTP 402.
• Every number in this post is printed by scripts/firewall-study.mts, run against that data.

131 sites say yes in robots.txt and no at the edge

Challenges outnumber hard refusals. A challenge page asks the client to prove it is a browser, which no crawler does, so in practice it stops the request just as a 403 would. It is also a typical answer from a bot manager that is unsure about a client, rather than one with a rule naming AI crawlers.

By CDN: counts are not rates

The raw counts would suggest the problem is spread across “no CDN”, Akamai and Cloudflare. Dividing by how many sites in the allow-all, clean-baseline group sit on each CDN tells a different story.

More than a third of Akamai-fronted sites in the group turned our crawler-named request away. Cloudflare and CloudFront sites did so at about the same rate as sites with no CDN we could detect. We can’t see the rule that fired, so we don’t know how many of the Akamai blocks are crawler verification working as intended and how many are rules that would stop the real crawler too.

By domain ending

Government sites stand out: 14 of 43 .gov sites that allow all three crawlers in robots.txt blocked our request. Small groups swing easily, so read the .gov and .uk rates as a prompt to look, not a measured trend.

The wider count: 242 sites when no rules could be read

The 131 only includes sites whose robots.txt we could parse. If you instead count every listed site where no AI search crawler is disallowed, including sites where we read no rules at all, the figure is 242 of 2,832. Of the extra 111, 82 refused robots.txt itself with a 403, 16 served HTML instead of text, 7 had no file, 4 timed out, 1 was empty and 1 returned a server error.

Under RFC 9309, 221 of the 242 (parsed rules, a 4xx, or an empty file) count as allowing crawlers. We keep the headline at 131 because a site that refuses its own robots.txt is already blocking at the edge, and a robots.txt “yes” there is only an inference from the protocol.

How to check your own site

Start with the same comparison we ran: one request as a browser, one as each crawler, and compare the status codes.

# As a browser
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36" https://example.com/

# As each AI search crawler (token only; use the full string from each vendor's docs if you prefer)
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-SearchBot" https://example.com/

1. If the browser gets 200 and a crawler gets 403, 429 or a challenge page, your edge treats that name differently. This is the same test we ran, with the same limit: it comes from your network, not the crawler’s.
2. Look in your CDN or WAF for the rule that fired: AI-crawler blocking, bot-fight or bot-management modes, custom rules on the User-Agent, and rate limits.
3. Check your server or CDN logs for requests from the published IP ranges (openai.com/searchbot.json, perplexity.com/perplexitybot.json) and the status they got. That is the only evidence of what the real crawler met.
4. Run the free check at /tools/ai-crawler-checker, which reads your robots.txt for each AI crawler and runs the browser-versus-crawler comparison for you.
5. If you work from an AI assistant or agent, the same checks are available through our MCP server, described at /agents.

Limits

• One test run, on 13 October 2026, from one connection. Edge rules change, and some bot managers vary their answer by network and time.
• We requested the home page only. A site may treat deeper pages differently.
• Our requests carried the crawler’s User-Agent but not its IP address, so this measures how the edge treats the name, not how it treats the real crawler.
• CDN detection reads response headers; “none detected” includes sites behind CDNs that don’t identify themselves.