Study · tested 13 October 2026

Which of the world's top websites block AI crawlers?

We read the robots.txt of the top 5,000 websites for 31 AI crawlers, then requested each home page as ChatGPT's, Claude's and Perplexity's search crawlers to see what the firewall did. Look up any of the 3,218 sites that answered. Not on the list? You'll go straight to a live audit.

How often each crawler is blocked

GPTBot (OpenAI training)

14.5%

OAI-SearchBot (ChatGPT search)

7.5%

ClaudeBot (Anthropic training)

13.3%

PerplexityBot (Perplexity search)

10.4%

Share of the 2,332 live sites with readable robots.txt rules that disallow the crawler.

Notable

Block all three AI search crawlers in robots.txt (164)

robots.txt allows an AI search crawler, the firewall turns it away (270)

Block all 7 AI training crawlers (96)

Every site, by rank

Ranks 1–500 (311 sites)

Ranks 501–1,000 (318 sites)

Ranks 1,001–1,500 (339 sites)

Ranks 1,501–2,000 (314 sites)

Ranks 2,001–2,500 (320 sites)

Ranks 2,501–3,000 (314 sites)

Ranks 3,001–3,500 (319 sites)

Ranks 3,501–4,000 (326 sites)

Ranks 4,001–4,500 (332 sites)

Ranks 4,501–5,000 (325 sites)

Sample: Tranco top 5,000 (list 64VKX), tested once on 13 October 2026 from one Indian fixed-line connection. Sites that didn't answer and 92 adult sites are left out. Method, caveats and the CSV (CC BY 4.0) are in the study.

Free to run. Pay only for the fixes you want.

Free

No login

Your score, every check's result, and the full fix for the top 3 issues.

★ Starter

$9 / ₹499 once

Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.

Pro

$29 / ₹1,499 / month

Full reports on every site, weekly alerts when AI access breaks, competitor compare.

Agencies: white-label widget, branded reports and lead capture on Agency.