Do sites ranked 1696–1813 block AI crawlers?

Do they block AI crawlers? These are the 61 sites ranked 1696–1813 in our study of the top 5,000 websites, each with what its robots.txt and firewall did to the ChatGPT, Claude and Perplexity search crawlers on 13 October 2026. Open a site for all 31 crawlers we test, including its AI crawler access in robots.txt.

• afribaba.com #1,696: ChatGPT, Claude and Perplexity couldn't be tested
• eepurl.com #1,698: the ChatGPT, Claude and Perplexity search crawlers all got in
• an1.com #1,699: the ChatGPT, Claude and Perplexity search crawlers all got in
• letras.mus.br #1,701: ChatGPT, Claude and Perplexity blocked at the firewall
• ico.org.uk #1,703: the ChatGPT, Claude and Perplexity search crawlers all got in
• jra.jp #1,707: the ChatGPT, Claude and Perplexity search crawlers all got in
• google.it #1,708: the ChatGPT, Claude and Perplexity search crawlers all got in
• google.nl #1,709: the ChatGPT, Claude and Perplexity search crawlers all got in
• takarakuji-official.jp #1,711: the ChatGPT, Claude and Perplexity search crawlers all got in
• site.com #1,714: the ChatGPT, Claude and Perplexity search crawlers all got in
• keirin.jp #1,717: ChatGPT, Claude and Perplexity blocked in robots.txt
• edna.ru #1,718: the ChatGPT, Claude and Perplexity search crawlers all got in
• googll.store #1,719: ChatGPT, Claude and Perplexity couldn't be tested
• xosodaiphat.com #1,721: the ChatGPT, Claude and Perplexity search crawlers all got in
• arionplay.com #1,726: ChatGPT, Claude and Perplexity couldn't be tested
• flashscore.mobi #1,727: the ChatGPT, Claude and Perplexity search crawlers all got in
• sportfilm800.com #1,728: ChatGPT, Claude and Perplexity blocked in robots.txt
• humoruniv.com #1,729: Claude and Perplexity got in; ChatGPT blocked in robots.txt
• dord.gov.in #1,731: ChatGPT, Claude and Perplexity blocked at the firewall
• kemkes.go.id #1,733: the ChatGPT, Claude and Perplexity search crawlers all got in
• cmu.edu #1,734: the ChatGPT, Claude and Perplexity search crawlers all got in
• goal7.co #1,735: the ChatGPT, Claude and Perplexity search crawlers all got in
• promiedos.com.ar #1,739: the ChatGPT, Claude and Perplexity search crawlers all got in
• indown.io #1,740: ChatGPT, Claude and Perplexity couldn't be tested
• digitallocker.gov.in #1,741: the ChatGPT, Claude and Perplexity search crawlers all got in
• ecnavi.jp #1,743: the ChatGPT, Claude and Perplexity search crawlers all got in
• tjsp.jus.br #1,744: the ChatGPT, Claude and Perplexity search crawlers all got in
• sarkariresult.com.cm #1,745: the ChatGPT, Claude and Perplexity search crawlers all got in
• manhwaweb.com #1,750: the ChatGPT, Claude and Perplexity search crawlers all got in
• hdhub4u.bi #1,752: the ChatGPT, Claude and Perplexity search crawlers all got in
• bancodevenezuela.com #1,753: the ChatGPT, Claude and Perplexity search crawlers all got in
• livescore.cz #1,754: the ChatGPT, Claude and Perplexity search crawlers all got in
• csc.gov.in #1,755: the ChatGPT, Claude and Perplexity search crawlers all got in
• ojogodobicho.com #1,757: ChatGPT, Claude and Perplexity couldn't be tested
• choiceadvantage.com #1,759: the ChatGPT, Claude and Perplexity search crawlers all got in
• xsmn.mobi #1,760: the ChatGPT, Claude and Perplexity search crawlers all got in
• salonboard.com #1,762: ChatGPT, Claude and Perplexity couldn't be tested
• thscore99.com #1,763: the ChatGPT, Claude and Perplexity search crawlers all got in
• minhngoc.net.vn #1,765: the ChatGPT, Claude and Perplexity search crawlers all got in
• betpawa.ug #1,766: the ChatGPT, Claude and Perplexity search crawlers all got in
• espn.com.ve #1,767: the ChatGPT, Claude and Perplexity search crawlers all got in
• xoso.com.vn #1,771: the ChatGPT, Claude and Perplexity search crawlers all got in
• packzy.com #1,772: the ChatGPT, Claude and Perplexity search crawlers all got in
• mobiledokan.com #1,774: Claude and Perplexity got in; ChatGPT blocked at the firewall
• kqbd.mobi #1,775: the ChatGPT, Claude and Perplexity search crawlers all got in
• kernel.org #1,776: the ChatGPT, Claude and Perplexity search crawlers all got in
• loteriasdominicanas.com #1,778: ChatGPT, Claude and Perplexity couldn't be tested
• census.gov.in #1,787: the ChatGPT, Claude and Perplexity search crawlers all got in
• albumaty.com #1,788: the ChatGPT, Claude and Perplexity search crawlers all got in
• smadex.com #1,790: the ChatGPT, Claude and Perplexity search crawlers all got in
• az24.vn #1,792: the ChatGPT, Claude and Perplexity search crawlers all got in
• tamildhool.tech #1,795: ChatGPT, Claude and Perplexity couldn't be tested
• loopia.se #1,796: the ChatGPT, Claude and Perplexity search crawlers all got in
• cbl.gov.ly #1,797: the ChatGPT, Claude and Perplexity search crawlers all got in
• linkmake.in #1,799: the ChatGPT, Claude and Perplexity search crawlers all got in
• blackberry.net #1,802: the ChatGPT, Claude and Perplexity search crawlers all got in
• egramswaraj.gov.in #1,807: the ChatGPT, Claude and Perplexity search crawlers all got in
• dpboss.tax #1,808: the ChatGPT, Claude and Perplexity search crawlers all got in
• loteriadehoy.com #1,811: the ChatGPT, Claude and Perplexity search crawlers all got in
• shutterstock.com #1,812: ChatGPT, Claude and Perplexity couldn't be tested
• satta-king-fast.com #1,813: the ChatGPT, Claude and Perplexity search crawlers all got in