SEO·AEO·GEO
Sign in
Run a free auditPricingDoes AI name you?For agenciesDone for youResearchFree toolsMethodologyAbout

Top websites that block AI search crawlers

Which top websites block AI search crawlers (Oct 2026)? In our study of the top 5,000 websites on 13 October 2026, these are the highest-ranked sites that block ChatGPT, Claude and Perplexity search crawlers in robots.txt or at the firewall, and those that block every AI training crawler. Each list shows the top 20 by rank; every site is on the rank pages.

Block all three AI search crawlers in robots.txt (162)

  • fbcdn.net #12: ChatGPT, Claude and Perplexity blocked in robots.txt
  • mail.ru #14: ChatGPT, Claude and Perplexity blocked in robots.txt
  • twitter.com #16: ChatGPT, Claude and Perplexity blocked in robots.txt
  • dzen.ru #17: ChatGPT, Claude and Perplexity blocked in robots.txt
  • amazon.com #27: ChatGPT, Claude and Perplexity blocked in robots.txt
  • x.com #48: ChatGPT, Claude and Perplexity blocked in robots.txt
  • pinterest.com #53: ChatGPT, Claude and Perplexity blocked in robots.txt
  • msn.com #77: ChatGPT, Claude and Perplexity blocked in robots.txt
  • baidu.com #79: ChatGPT, Claude and Perplexity blocked in robots.txt
  • yandex.ru #98: ChatGPT, Claude and Perplexity blocked in robots.txt
  • reddit.com #104: ChatGPT, Claude and Perplexity blocked in robots.txt
  • registrar-servers.com #120: ChatGPT, Claude and Perplexity blocked in robots.txt
  • nytimes.com #155: ChatGPT, Claude and Perplexity blocked in robots.txt
  • cnn.com #198: ChatGPT, Claude and Perplexity blocked in robots.txt
  • t.co #241: ChatGPT, Claude and Perplexity blocked in robots.txt
  • imdb.com #263: ChatGPT, Claude and Perplexity blocked in robots.txt
  • launchpad.net #278: ChatGPT, Claude and Perplexity blocked in robots.txt
  • skybridge.click #279: ChatGPT, Claude and Perplexity blocked in robots.txt
  • amazon.co.uk #332: ChatGPT, Claude and Perplexity blocked in robots.txt
  • amazon.de #402: ChatGPT, Claude and Perplexity blocked in robots.txt

robots.txt allows an AI search crawler, the firewall turns it away (262)

  • digicert.com #52: ChatGPT got in; Claude and Perplexity blocked at the firewall
  • adobe.com #63: ChatGPT, Claude and Perplexity blocked at the firewall
  • endpoints.news #65: Claude got in; ChatGPT and Perplexity blocked at the firewall
  • chatgpt.com #90: Perplexity blocked in robots.txt; ChatGPT and Claude blocked at the firewall
  • wordpress.com #100: ChatGPT, Claude and Perplexity blocked at the firewall
  • openai.com #110: ChatGPT, Claude and Perplexity blocked at the firewall
  • intuit.com #132: Claude and Perplexity got in; ChatGPT blocked at the firewall
  • tumblr.com #154: Claude and Perplexity got in; ChatGPT blocked at the firewall
  • unity3d.com #161: ChatGPT, Claude and Perplexity blocked at the firewall
  • webex.com #162: ChatGPT, Claude and Perplexity blocked at the firewall
  • xiaomi.com #178: ChatGPT, Claude and Perplexity blocked at the firewall
  • medium.com #197: ChatGPT, Claude and Perplexity blocked at the firewall
  • forbes.com #216: Claude got in; Perplexity blocked in robots.txt; ChatGPT blocked at the firewall
  • cisco.com #228: ChatGPT, Claude and Perplexity blocked at the firewall
  • meraki.com #243: ChatGPT, Claude and Perplexity blocked at the firewall
  • ibm.com #274: ChatGPT, Claude and Perplexity blocked at the firewall
  • pubmatic.com #289: Claude got in; ChatGPT and Perplexity blocked at the firewall
  • springer.com #297: ChatGPT and Claude got in; Perplexity blocked at the firewall
  • autodesk.com #300: ChatGPT, Claude and Perplexity blocked at the firewall
  • salesforce.com #305: ChatGPT, Claude and Perplexity blocked at the firewall

Block all 7 AI training crawlers (94)

  • fbcdn.net #12: ChatGPT, Claude and Perplexity blocked in robots.txt
  • mail.ru #14: ChatGPT, Claude and Perplexity blocked in robots.txt
  • twitter.com #16: ChatGPT, Claude and Perplexity blocked in robots.txt
  • dzen.ru #17: ChatGPT, Claude and Perplexity blocked in robots.txt
  • linkedin.com #18: ChatGPT and Claude got in; Perplexity blocked in robots.txt
  • x.com #48: ChatGPT, Claude and Perplexity blocked in robots.txt
  • pinterest.com #53: ChatGPT, Claude and Perplexity blocked in robots.txt
  • baidu.com #79: ChatGPT, Claude and Perplexity blocked in robots.txt
  • yandex.ru #98: ChatGPT, Claude and Perplexity blocked in robots.txt
  • reddit.com #104: ChatGPT, Claude and Perplexity blocked in robots.txt
  • registrar-servers.com #120: ChatGPT, Claude and Perplexity blocked in robots.txt
  • flickr.com #184: the ChatGPT, Claude and Perplexity search crawlers all got in
  • t.co #241: ChatGPT, Claude and Perplexity blocked in robots.txt
  • imdb.com #263: ChatGPT, Claude and Perplexity blocked in robots.txt
  • skybridge.click #279: ChatGPT, Claude and Perplexity blocked in robots.txt
  • reuters.com #282: Claude and Perplexity blocked in robots.txt; ChatGPT couldn't be tested
  • threads.com #594: ChatGPT, Claude and Perplexity blocked in robots.txt
  • imgsmail.ru #651: ChatGPT, Claude and Perplexity blocked in robots.txt
  • sharethrough.com #663: ChatGPT, Claude and Perplexity blocked in robots.txt
  • yelp.com #709: ChatGPT, Claude and Perplexity blocked in robots.txt