India AI crawler index: which .in sites block ChatGPT, Claude and Perplexity?
We read the robots.txt of the 500 highest-ranked commercial .in sites and the 100 highest-ranked government sites, and worked out what each one tells 31 AI crawlers. Blocking a training crawler keeps a site out of model training; blocking a search crawler keeps it out of that AI's answers. Here is who does which, next to our global top-5,000 numbers.
Commercial .in sites
500 sites sampled; 332 served readable robots.txt rules. Didn't answer: 75 · No robots.txt (404): 53 · Server error: 7 · Empty file: 2 · Refused (401/403): 16 · HTTP 418: 1 · HTML page instead of rules: 14.
| Crawler | Indian sites blocking it | Global top 5,000 |
|---|---|---|
| GPTBot (OpenAI training) | 9.9% | 14.5% |
| OAI-SearchBot (ChatGPT search) | 3.6% | 7.5% |
| ClaudeBot (Anthropic training) | 9.6% | 13.3% |
| Claude-SearchBot (Claude search) | 4.8% | 7.1% |
| PerplexityBot (Perplexity search) | 4.8% | 10.4% |
| Google-Extended (Gemini training) | 8.1% | 11.7% |
| CCBot (Common Crawl) | 10.2% | 15.1% |
| Bytespider (ByteDance) | 12.0% | 15.0% |
Share of the 332 sites above with readable rules that disallow the crawler. The global column is the share of live sites with readable rules in our 13 October top-5,000 study.
Block all three AI search crawlers in robots.txt (12)
- amazon.in
- ocbnyutpfwjs.in
- adjust.net.in
- lite-host.in
- two-buddy.in
- qlc.co.in
- growthrx.in
- website999.co.in
- pinboard.in
- divyabhaskar.co.in
- dreamplug.in
- filmai.in
Block one or two of them (6)
Block all 7 AI training crawlers (13)
- lnkd.in
- ocbnyutpfwjs.in
- adjust.net.in
- lite-host.in
- two-buddy.in
- qlc.co.in
- growthrx.in
- website999.co.in
- pinboard.in
- redecanais.in
- dreamplug.in
- filmai.in
- kbuses.in
Every commercial site that answered, by rank (403)
- amazon.in
- google.co.in
- aajtak.in
- goodreturns.in
- redbus.in
- screener.in
- linkmake.in
- educationportal3.in
- lnkd.in
- registry.in
- indiatoday.in
- airtel.in
- goindigo.in
- groww.in
- runnr.in
- bajajfinserv.in
- ctrls.in
- olx.in
- irctc.co.in
- zoho.in
- nobroker.in
- livescore.in
- porter.in
- mxplayer.in
- hpcl.co.in
- district.in
- npci.org.in
- ernet.in
- cashify.in
- skyscanner.co.in
- ndtv.in
- railyatri.in
- apollopharmacy.in
- tripadvisor.in
- reliancedigital.in
- decathlon.in
- tanishq.co.in
- lpu.in
- proteantech.in
- titan.co.in
- allahabadhighcourt.in
- learncbse.in
- ipowatch.in
- tatasteel.co.in
- poshantracker.in
- faymas.in
- dpbosss.net.in
- tnpscexams.in
- ocbnyutpfwjs.in
- majhinaukri.in
- callbomberz.in
- manakonline.in
- pmvidyalaxmi.co.in
- lotterysambadresult.in
- sattakingy.in
- storytv.co.in
- kolkataff.in
- mahateacherrecruitment.org.in
- qbrik.in
- shillongteergrounds.in
- showpm.in
- teletype.in
- blogspot.in
- sweatco.in
- angelone.in
- adjust.net.in
- allevents.in
- glassdoor.co.in
- businesstoday.in
- theprint.in
- rbi.org.in
- gettyimages.in
- hostgator.in
- strip2.in
- brainly.in
- foundit.in
- thewire.in
- crackle.co.in
- aninews.in
- dailyhunt.in
- juspay.in
- nkn.in
- lite-host.in
- licindia.in
- thebase.in
- pharmeasy.in
- google.in
- indiatv.in
- logme.in
- allen.in
- y99.in
- boyfriendtv.in
- valmo.in
- apcfss.in
- shadowfax.in
- cuchd.in
- mahadiscom.in
- altin.in
- canva.in
- popt.in
- pspcl.in
- zapvi.in
- guidely.in
- 4raicefishpwa.in
- oliveboard.in
- libas.in
- deodap.in
- jockey.in
- fastrack.in
- alphacommunity.in
- bokepud.in
- baji999.in
- ksrtc.in
- futureskillsprime.in
- oziva.in
- tnstc.in
- betswiz.in
- zardfilm.in
- offduty.in
- shiprocket.in
- trackon.in
- indiahood.in
- gsrtc.in
- bokepnoz.in
- desipussy.in
- nikshay.in
- uiic.in
- nagalandstatelottery.in
- scroll.in
- magicpin.in
- kredito24.in
- storymax.in
- saveetha.in
- filmyzilla60.in
- beyondappliances.in
- castle.am.in
- dearlottery.in
- gebyartrivora.io.in
- invisalign.in
- matka420.in
- niconline.co.in
- pddrc.in
- two-buddy.in
- bigrock.in
- actcorp.in
- airbnb.co.in
- netplus.co.in
- theweek.in
- intoday.in
- reqres.in
- audible.in
- ubuy.co.in
- wildla.in
- christuniversity.in
- mygov.in
- freepressjournal.in
- garv.in
- espn.in
- vogue.in
- flashscore.in
- downtoearth.org.in
- hostingraja.in
- dwan.co.in
- seair.co.in
- digit.in
- payu.in
- cadburygifting.in
- grabweb.in
- qlc.co.in
- oneplus.in
- businessworld.in
- freelancer.in
- livelaw.in
- growthrx.in
- zapf.in
- website999.co.in
- goldclub.co.in
- fusionnet.in
- wttr.in
- cleartax.in
- bbc.in
- adsolut.in
- chrono24.in
- ultranet.co.in
- workindia.in
- dominos.co.in
- mgovcloud.in
- stpi.in
- logix.in
- aqi.in
- bsnl.co.in
- cdac.in
- fyers.in
- transfermarkt.co.in
- kseb.in
- planetspark.in
- serverbyt.in
- pinboard.in
- edurev.in
- medicoverhospitals.in
- parkpage.in
- desertcart.in
- youbroadband.in
- insider.in
- dnshostserver.in
- indiafreestuff.in
- mojapp.in
- digikey.in
- instamart.in
- vikaspedia.in
- ibtimes.co.in
- maxhealthcare.in
- worldphone.in
- myvi.in
- sdbi.in
- tamilchristiansongs.in
- tooter.in
- shophost.in
- myg.in
- bankofbaroda.in
- simplyhired.co.in
- robu.in
- equal.in
- gktoday.in
- dmart.in
- expedia.co.in
- wynk.in
- sony.co.in
- directverify.in
- starhealth.in
- baseconnect.in
- tradebrains.in
- finbox.in
- coalindia.in
- sbigeneral.in
- grabon.in
- sbilife.co.in
- truemeds.in
- antiragging.in
- hostinger.in
- divyabhaskar.co.in
- prepp.in
- maxlab.co.in
- sangbadpratidin.in
- sheinindia.in
- orra.co.in
- newindia.co.in
- platinumrx.in
- dyson.in
- onlinebakra.in
- er4ustartup.in
- nooks.in
- adidas.co.in
- indianoil.co.in
- digihaat.in
- mediassist.in
- architecturaldigest.in
- nixiregistry.in
- bajajfinservmarkets.in
- qualys.in
- heraldgoa.in
- kreditbee.in
- adgebra.co.in
- sbi.co.in
- uhbvn.org.in
- kapiva.in
- easebuzz.in
- xbees.in
- visionias.in
- m440.in
- vyaparapp.in
- bharatpetroleum.in
- taxguru.in
- indiatodayne.in
- wbsedcl.in
- careerpower.in
- hindutamil.in
- winni.in
- indianbank.in
- nike.in
- visa.co.in
- konkur.in
- guruvayoorlive.in
- newlotus365.co.in
- epson.co.in
- shriramfinance.in
- nestasia.in
- levi.in
- varthabharati.in
- esic.in
- redecanais.in
- sbpdcl.co.in
- schools.org.in
- dnspark.in
- nasscom.in
- dreamplug.in
- mybillbook.in
- mintindia.in
- icmai.in
- erail.in
- sansad.in
- ckycindia.in
- vaku.in
- mamaearth.in
- paradice.in
- nbpdcl.co.in
- shuru.co.in
- sbisecurities.in
- kluniversity.in
- airtelxstream.in
- motorola.in
- hmtwatches.in
- marksandspencer.in
- yourprint.in
- orientalinsurance.org.in
- muftijeans.in
- bankofbaroda.co.in
- hopscotch.in
- mgmotor.co.in
- moneyview.in
- cracku.in
- bseh.org.in
- grse.in
- biba.in
- pobb.in
- epw.in
- salty.co.in
- mockers.in
- magmatech.in
- sarimusic.in
- adzuna.in
- publictv.in
- rubans.in
- licious.in
- countrydelight.in
- cris.org.in
- aajkaal.in
- headphonezone.in
- homecredit.co.in
- gokiwi.in
- vistaprint.in
- memes.co.in
- limanowa.in
- restaurant-guru.in
- renault.co.in
- appx.co.in
- designinfo.in
- ramrajcotton.in
- digio.in
- apsrtconline.in
- nakkheeran.in
- icar.org.in
- nvmessd-servers.in
- volkswagen.co.in
- justintime.in
- thesleepcompany.in
- filmai.in
- govtschemes.in
- suzukimotorcycle.co.in
- milton.in
- technosport.in
- vplink.in
- autodraft.in
- nnyy.in
- jansamarth.in
- jackjones.in
- darwinbox.in
- guardian.in
- pincode.net.in
- yourbus.in
- luciferdonghua.in
- behmelody.in
- kbuses.in
- edcil.co.in
- randstad.in
- amway.in
- quackquack.in
- apollodiagnostics.in
- tpsl-india.in
- skechers.in
- bizom.in
- joyalukkas.in
- nissan.in
- gipl.in
- hpushimla.in
- foxtale.in
- gameloot.in
- tactv.in
- paiinternational.in
- crocs.in
- dmifinance.in
- iibf.org.in
- ipopremium.in
- fiberx.co.in
- uudoon.in
- findauction.in
- 4rapwavi.in
- apnacollege.in
- zouk.co.in
- eauto.co.in
- realbooks.in
- uspoloassn.in
- railwire.co.in
Government sites (.gov.in and .nic.in)
100 sites sampled; 33 served readable robots.txt rules. HTML page instead of rules: 15 · Didn't answer: 16 · No robots.txt (404): 30 · Refused (401/403): 5 · Server error: 1.
| Crawler | Indian sites blocking it | Global top 5,000 |
|---|---|---|
| GPTBot (OpenAI training) | 12.1% | 14.5% |
| OAI-SearchBot (ChatGPT search) | 12.1% | 7.5% |
| ClaudeBot (Anthropic training) | 12.1% | 13.3% |
| Claude-SearchBot (Claude search) | 12.1% | 7.1% |
| PerplexityBot (Perplexity search) | 12.1% | 10.4% |
| Google-Extended (Gemini training) | 12.1% | 11.7% |
| CCBot (Common Crawl) | 12.1% | 15.1% |
| Bytespider (ByteDance) | 12.1% | 15.0% |
Share of the 33 sites above with readable rules that disallow the crawler. The global column is the share of live sites with readable rules in our 13 October top-5,000 study.
Block all three AI search crawlers in robots.txt (4)
Block one or two of them (0)
None.
Block all 7 AI training crawlers (4)
Every government site that answered, by rank (83)
- rajasthan.gov.in
- gst.gov.in
- karnataka.gov.in
- incometax.gov.in
- uidai.gov.in
- eci.gov.in
- wb.gov.in
- epfindia.gov.in
- up.gov.in
- indianrail.gov.in
- parivahan.gov.in
- dord.gov.in
- digitallocker.gov.in
- csc.gov.in
- census.gov.in
- egramswaraj.gov.in
- telangana.gov.in
- maharashtra.gov.in
- tn.gov.in
- mca.gov.in
- mohfw.gov.in
- jharkhand.gov.in
- delhi.gov.in
- education.gov.in
- indiapost.gov.in
- mponline.gov.in
- esic.gov.in
- digilocker.gov.in
- gem.gov.in
- swayam.gov.in
- cybercrime.gov.in
- scholarships.gov.in
- pib.gov.in
- ap.gov.in
- lsgkerala.gov.in
- nha.gov.in
- udyamregistration.gov.in
- pgportal.gov.in
- haryana.gov.in
- wcd.gov.in
- udiseplus.gov.in
- mybharat.gov.in
- aranya.gov.in
- samsodisha.gov.in
- meripehchaan.gov.in
- cisf.gov.in
- kvs.gov.in
- upbhulekh.gov.in
- samagra.gov.in
- up.nic.in
- pdsodisha.gov.in
- ncert.nic.in
- igrsup.gov.in
- nrega.nic.in
- admissions.nic.in
- examinationservices.nic.in
- joinindianarmy.nic.in
- india.gov.in
- indianvisaonline.gov.in
- passportindia.gov.in
- indianrailways.gov.in
- ecourts.gov.in
- odisha.gov.in
- assam.gov.in
- fssai.gov.in
- upsc.gov.in
- sci.gov.in
- cbse.gov.in
- cgg.gov.in
- ssc.gov.in
- abdm.gov.in
- umang.gov.in
- rrbapply.gov.in
- ncrb.gov.in
- rtionline.gov.in
- sancharsaathi.gov.in
- epfo.gov.in
- skillindiadigital.gov.in
- pmaymis.gov.in
- eshram.gov.in
- agristack.gov.in
- mahabhumi.gov.in
- patnahighcourt.gov.in
Run an Indian site?
To stay in ChatGPT, Claude and Perplexity answers while opting out of training, build your rules with the AI robots.txt generator. robots.txt is only half of it: a CDN or firewall can still turn AI crawlers away, which a free live audit tests. For an Indian-rupee report and fixes, see the AI visibility audit for India.
Method: the 500 highest-ranked .in domains on Tranco list 64VKX that aren't .gov.in, .nic.in, .ac.in, .edu.in, .res.in, .mil.in or .ernet.in (adult domains left out), plus the 100 highest-ranked .gov.in and .nic.in domains. Checked once on 20 October 2026 from one Indian fixed-line connection with a browser user agent; robots.txt parsed per RFC 9309, as in our audit. robots.txt only: unlike the global study, there are no firewall tests, so a site that allows a crawler here may still block it at the edge. Indian brands on .com (and short links or CDN hosts on .in) are not separated out, so this is a sample of the .in namespace, not of Indian business as a whole. Block rates use only sites with readable rules. Data: CSV (CC BY 4.0).
Free to run. Pay only for the fixes you want.
Free
No login
Your score, every check's result, and the full fix for the top 3 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.