AI crawler · Anthropic · Model training
ClaudeBot: what it is, its user agent, and how to block it
ClaudeBot is Anthropic's AI training crawler. Collects web content that may be used to train Anthropic's models. In our 13 October 2026 study, 310 of 2,332 live top sites with readable robots.txt rules (13.3%) disallow ClaudeBot. Part of the AI crawler list.
What ClaudeBot does
Operator: Anthropic. Job: Model training. Obeys robots.txt.
“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.”
Anthropic's ClaudeBot documentationPublished IP ranges (JSON)
ClaudeBot user agent
robots.txt token: ClaudeBot (matched case-insensitively).
Anthropic doesn't publish a full user-agent string for ClaudeBot. Look for ClaudeBot in the User-Agent header.
robots.txt: allow or block ClaudeBot
Allow
User-agent: ClaudeBot
Allow: /Block
User-agent: ClaudeBot
Disallow: /Training only, not citations. Blocking it is a training opt-out. It doesn't decide whether Anthropic's AI answers can cite you; Claude-SearchBot does that, and it has its own robots.txt group.
Once a group names ClaudeBot, it ignores your User-agent: * rules, so copy any paths you still want blocked into it.
How many top sites block ClaudeBot
Disallow ClaudeBot in robots.txt
13.3%
310 of 2,332 sites
In our 13 October 2026 study, 310 of 2,332 live top sites with readable robots.txt rules (13.3%) disallow ClaudeBot.
Sample: Tranco top 5,000, tested once on 13 October 2026 from one Indian fixed-line connection. The robots.txt share counts live sites (home page 2xx) whose rules we could read, as on the AI visibility hub. Method and CSV in the study.
Check your own site
The AI crawler checker reads your robots.txt the way ClaudeBot does and tests what your firewall returns. To write the rules, use the AI robots.txt generator.
Check your page: free, about 20 seconds, no sign-in
What it checks
- robots.txt for ClaudeBot
- Finds the group ClaudeBot obeys under RFC 9309 and the rule that decides your page.
- What your CDN lets through
- Requests the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compares the answers with a browser request.
- Content without JavaScript
- AI crawlers don't run JavaScript. The audit checks how much of the page is in the raw HTML.
Free to run. Pay only for the fixes you want.
Free
Free account
Your score without signing in; with a free account, every check's result and the full fix for the top 1 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.
Questions
What is ClaudeBot?+
ClaudeBot is Anthropic's AI training crawler. Collects web content that may be used to train Anthropic's models.
What is ClaudeBot's user agent?+
Anthropic publishes only the token ClaudeBot. Match on it in the User-Agent header.
How do I block ClaudeBot?+
Add "User-agent: ClaudeBot" followed by "Disallow: /" to your robots.txt. Obeys robots.txt, per Anthropic.
Does blocking ClaudeBot affect AI search citations?+
Blocking it is a training opt-out. It doesn't decide whether Anthropic's AI answers can cite you; Claude-SearchBot does that, and it has its own robots.txt group.
How many websites block ClaudeBot?+
In our 13 October 2026 study, 310 of 2,332 live top sites with readable robots.txt rules (13.3%) disallow ClaudeBot.