Does lite-host.in block AI crawlers?
lite-host.in's robots.txt blocked all three AI search crawlers (OAI-SearchBot, Claude-SearchBot and PerplexityBot). It blocked 7 of the 7 AI training crawlers we track. We read its robots.txt on 20 October 2026 for our India AI crawler index of commercial .in sites; it ranks #17,057 on the Tranco list.
What robots.txt tells AI search crawlers
OAI-SearchBot
Blocked in robots.txt
Blocked (rule for all crawlers).
Claude-SearchBot
Blocked in robots.txt
Blocked (rule for all crawlers).
PerplexityBot
Blocked in robots.txt
Blocked (rule for all crawlers).
This index reads robots.txt only. A CDN or firewall can still turn a crawler away that robots.txt allows; a live audit tests that.
This is a snapshot from 20 October 2026.
Run a live audit to see what AI crawlers get from lite-host.in today, firewall included, plus 50 more checks.
At a glance
- AI training crawlers blocked
- 7 of 7 (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, MistralAI-Training, Bytespider, CCBot)
- robots.txt
- Published, with rules
- Home page to a browser
- HTTP 200
- Segment
- Commercial .in
All 31 crawlers in lite-host.in's robots.txt
| Crawler | Job | lite-host.in | Indian commercial sites blocking it |
|---|---|---|---|
| OAI-SearchBot | AI search index | Blocked (rule for all crawlers) | 3.6% |
| Claude-SearchBot | AI search index | Blocked (rule for all crawlers) | 4.8% |
| PerplexityBot | AI search index | Blocked (rule for all crawlers) | 4.8% |
| MistralAI-Index | AI search index | Blocked (rule for all crawlers) | 3.9% |
| YouBot | AI search index | Blocked (rule for all crawlers) | 6.3% |
| ChatGPT-User | Fetches when a user asks | Blocked (rule for all crawlers) | 5.4% |
| Claude-User | Fetches when a user asks | Blocked (rule for all crawlers) | 5.7% |
| Perplexity-User | Fetches when a user asks | Blocked (rule for all crawlers) | 5.7% |
| Google-GeminiNotebook | Fetches when a user asks | Blocked (rule for all crawlers) | 4.2% |
| Meta-ExternalFetcher | Fetches when a user asks | Blocked (rule for all crawlers) | 5.7% |
| Amzn-User | Fetches when a user asks | Blocked (rule for all crawlers) | 4.2% |
| DuckAssistBot | Fetches when a user asks | Blocked (rule for all crawlers) | 5.4% |
| MistralAI-User | Fetches when a user asks | Blocked (rule for all crawlers) | 5.4% |
| Googlebot | Search engine | Blocked (rule for all crawlers) | 2.4% |
| Bingbot | Search engine | Blocked (rule for all crawlers) | 2.7% |
| Meta-WebIndexer | Search engine | Blocked (rule for all crawlers) | 5.4% |
| Amzn-SearchBot | Search engine | Blocked (rule for all crawlers) | 4.2% |
| Applebot | Search and training under one name | Blocked (rule for all crawlers) | 3.0% |
| Meta-ExternalAgent | Search and training under one name | Blocked (rule for all crawlers) | 8.7% |
| Amazonbot | Search and training under one name | Blocked (rule for all crawlers) | 9.0% |
| GPTBot | Model training | Blocked (rule for all crawlers) | 9.9% |
| ClaudeBot | Model training | Blocked (rule for all crawlers) | 9.6% |
| Google-Extended | Model training | Blocked (rule for all crawlers) | 8.1% |
| Applebot-Extended | Model training | Blocked (rule for all crawlers) | 8.4% |
| MistralAI-Training | Model training | Blocked (rule for all crawlers) | 4.8% |
| Bytespider | Model training | Blocked (rule for all crawlers) | 12% |
| CCBot | Model training | Blocked (rule for all crawlers) | 10% |
| Google-Agent | Browses and acts for a user | Blocked (rule for all crawlers) | 4.2% |
| OAI-AdsBot | Ads checks and site-owner services | Blocked (rule for all crawlers) | 3.9% |
| Google-CloudVertexBot | Ads checks and site-owner services | Blocked (rule for all crawlers) | 6.0% |
| Meta-ExternalAds | Ads checks and site-owner services | Blocked (rule for all crawlers) | 3.9% |
The last column is the share of sites in the same segment of the index with readable robots.txt rules. Matching follows RFC 9309: a crawler obeys the group that names it, otherwise the rule for all crawlers.
Run lite-host.in?
Blocking training crawlers doesn't remove you from AI answers; blocking the search crawlers does. To keep AI search and opt out of training, use the AI robots.txt generator, then check your CDN with a live audit. Every crawler's user agent is in the AI crawler list.
Free to run. Pay only for the fixes you want.
Free
No login
Your score, every check's result, and the full fix for the top 3 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.