Free tool · Cloudflare AI crawl settings

Is Cloudflare blocking ChatGPT or Googlebot on your site?

Since 15 September 2026, Cloudflare sorts crawlers into Search, Training and Agent, and only two settings, Block and Block on pages with ads, stop mixed-use crawlers like Googlebot, Bingbot and Applebot. Enter a URL: we detect Cloudflare, read your managed robots.txt and Content-Signal line, then request the page as OAI-SearchBot, Claude-SearchBot and PerplexityBot and compare each answer with a browser's. Background: what each Cloudflare setting does now

Free · no login · results in about 20 seconds

What it checks

Is the site on Cloudflare: Detected from the response headers, so the rest of the report knows which settings could apply.
Managed robots.txt and Content-Signal: Whether Cloudflare is writing your robots.txt, and what its search, ai-input and ai-train signals say. ai-train=no is what Disallow AI Training publishes, and that setting spares Googlebot.
Live requests as AI search crawlers: The page fetched as OAI-SearchBot, Claude-SearchBot and PerplexityBot next to a browser request: 403s, HTTP 402 pay-per-crawl, cf-mitigated challenge pages and suspiciously small bodies.
robots.txt per crawler: What your rules say to each search, AI-search, user-triggered and training crawler for this exact URL, kept separate so a training block isn't mistaken for lost visibility.
Content without JavaScript: Even with every crawler allowed, ChatGPT's, Claude's and Perplexity's crawlers read only the HTML your server sends.

Check it yourself

Replace example.com with your domain.

1. Is Cloudflare in front, and is it challenging the crawler?

A cf-mitigated: challenge header, or a 403 for the crawler but 200 for the browser, means the edge treats it differently.

curl -s -D - -o /dev/null -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/ | grep -i -E "^(HTTP|server|cf-mitigated|cf-ray)"
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://example.com/

2. Read the managed robots.txt and Content-Signal

Look for Content-Signal: search=yes, ai-train=no and for User-agent groups Cloudflare added.

curl -s https://example.com/robots.txt | grep -i -E "content-signal|user-agent|disallow" | head -40

Free to run. Pay only for the fixes you want.

Free

No login

Your score, every check's result, and the full fix for the top 3 issues.

★ Starter

$9 / ₹499 once

Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.

Pro

$29 / ₹1,499 / month

Full reports on every site, weekly alerts when AI access breaks, competitor compare.

Agencies: white-label widget, branded reports and lead capture on Agency.

Questions

Did Cloudflare start blocking Googlebot on 15 September?+

Not by itself. Every training block that existed before that date was migrated to Disallow AI Training, which keeps Googlebot, Bingbot and Applebot. A zone refuses them now only if someone chose Block or Block on pages with ads afterwards, or blocked Search.

Why might ChatGPT still fail to read my pages?+

The preset Cloudflare offers new ad-funded domains sets Agent to Block on pages with ads, which can stop ChatGPT fetching a page a user asked about. Bot Fight Mode and custom WAF rules can also challenge AI crawlers regardless of these settings.

Is your test the same as the real crawler?+

No. Real crawlers are verified by IP address, reverse DNS or a Web Bot Auth signature, and Cloudflare can treat a request that only uses a crawler's name differently. A block we see is a strong warning to check your settings; a pass is good evidence, not proof.

What should I set to stay in AI search but out of training?+

Training: Disallow AI Training. Search: Allow. Agent: Allow if you want assistants to read pages people ask about. Then run this check again to confirm the edge agrees with your settings.

What's free and what's paid?+

The check, the score and the full fix for the top 3 issues are free with no login. Starter unlocks every fix for one site for 30 days with one payment; Pro covers every site and alerts you if access breaks later.