AI crawler · OpenAI · AI search index

OAI-SearchBot: what it is, its user agent, and how to block it

OAI-SearchBot is OpenAI's AI search crawler. Builds the index ChatGPT search uses to show and link websites in its answers. In our 13 October 2026 study, 175 of 2,332 live top sites with readable robots.txt rules (7.5%) disallow OAI-SearchBot. Part of the AI crawler list.

What OAI-SearchBot does

Operator: OpenAI. Job: AI search index. Obeys robots.txt.

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below.”

OpenAI's OAI-SearchBot documentation Published IP ranges (JSON)

OAI-SearchBot user agent

robots.txt token: OAI-SearchBot (matched case-insensitively).

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

OpenAI calls this an example: the version number may change. A variant with a robots.txt marker fetches robots.txt.

robots.txt: allow or block OAI-SearchBot

Allow

User-agent: OAI-SearchBot
Allow: /

Block

User-agent: OAI-SearchBot
Disallow: /

Affects AI answers and citations. Blocking it keeps your pages out of the index OpenAI's AI search answers cite. It is not a training crawler.

Once a group names OAI-SearchBot, it ignores your User-agent: * rules, so copy any paths you still want blocked into it.

How many top sites block OAI-SearchBot

Disallow OAI-SearchBot in robots.txt

7.5%

175 of 2,332 sites

Allowed by robots.txt, refused at the firewall

8.4%

221 of 2,644 sites

In our 13 October 2026 study, 175 of 2,332 live top sites with readable robots.txt rules (7.5%) disallow OAI-SearchBot. Of the 2,644 listed sites whose robots.txt allowed OAI-SearchBot and that answered a browser cleanly, 221 (8.4%) refused our OAI-SearchBot request at the firewall with an error or challenge page.

Sample: Tranco top 5,000, tested once on 13 October 2026 from one Indian fixed-line connection. The robots.txt share counts live sites (home page 2xx) whose rules we could read, as on the AI visibility hub. Firewall tests came from our server, not the crawler's published IP ranges, so a refusal can also be an edge that verifies crawlers by IP. Method and CSV in the study.

Check your own site

The AI crawler checker reads your robots.txt the way OAI-SearchBot does and tests what your firewall returns. To write the rules, use the AI robots.txt generator.

Check your page: free, about 20 seconds, no sign-in

What it checks

robots.txt for OAI-SearchBot: Finds the group OAI-SearchBot obeys under RFC 9309 and the rule that decides your page.
What your CDN lets through: Requests the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compares the answers with a browser request.
Content without JavaScript: AI crawlers don't run JavaScript. The audit checks how much of the page is in the raw HTML.

Free to run. Pay only for the fixes you want.

Free

Free account

Your score without signing in; with a free account, every check's result and the full fix for the top 1 issues.

★ Starter

$9 / ₹499 once

Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.

Pro

$29 / ₹1,499 / month

Full reports on every site, weekly alerts when AI access breaks, competitor compare.

Agencies: white-label widget, branded reports and lead capture on Agency.

Questions

What is OAI-SearchBot?+

OAI-SearchBot is OpenAI's AI search crawler. Builds the index ChatGPT search uses to show and link websites in its answers.

What is OAI-SearchBot's user agent?+

OpenAI documents it as: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Match on the token OAI-SearchBot, since version numbers change.

How do I block OAI-SearchBot?+

Add "User-agent: OAI-SearchBot" followed by "Disallow: /" to your robots.txt. Obeys robots.txt, per OpenAI.

Does blocking OAI-SearchBot affect AI search citations?+

Blocking it keeps your pages out of the index OpenAI's AI search answers cite. It is not a training crawler.

How many websites block OAI-SearchBot?+