Meta-WebIndexer, Meta-ExternalFetcher and Amzn-SearchBot: The New AI Crawlers in Your Logs, and Whether to Allow Them

Meta-WebIndexer is the crawler that builds Meta AI's search index, and Amzn-SearchBot does the same job for Amazon's search experiences such as Alexa. Both are search bots: Meta says allowing Meta-WebIndexer “helps us cite and link to your content in Meta AI's responses”, and Amazon says Amzn-SearchBot “does not crawl content for generative AI model training”. Their siblings do different jobs. Meta-ExternalAgent may train models, Amazonbot “may be used to train Amazon AI models”, and the user-triggered fetchers (Meta-ExternalFetcher and Amzn-User) may ignore robots.txt. If you want to be cited by Meta AI and Alexa, allow the two search bots by name and treat the training bots as a separate decision.

This page lists the exact strings you will see in your logs, quotes what each operator says about robots.txt, shows what 39 large sites did with these bots on 13 October 2026, and ends with copy-paste robots.txt groups. If you are new to the search-versus-training split, GPTBot vs OAI-SearchBot explains it for OpenAI.

The exact user agents in your logs

Two details trip people up. Meta's strings carry a relative path, (+/documentation/…), not a full URL; that is how Meta publishes them, so don't treat it as a sign of a fake. And Amazon documents version 0.1 for all three of its bots. If a log line shows another version, the token is what robots.txt rules match on, and the source IP is what proves who sent it.

# Count requests per Meta and Amazon bot in an access log (adjust the path)
grep -i -o -E 'meta-(webindexer|externalagent|externalfetcher|externalads)|facebookexternalhit|amzn-(searchbot|user)|amazonbot' /var/log/nginx/access.log | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

What each bot does, in the operators' words

The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.

Amzn-SearchBot is used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa.

• Meta-ExternalAgent “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” That makes it mixed-use, not purely training.
• Meta-ExternalFetcher “fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users.”
• Meta-ExternalAds “crawls the web for use cases such as improving advertising and other business-related products and services.”
• Amazonbot “is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.”
• Amzn-User “supports user actions, such as responding to Alexa queries that require up-to-date information” and, like Amzn-SearchBot, “does not crawl content for generative AI model training.”

Amazonbot is the one most guides get wrong. It is often listed as Amazon's search crawler, but Amazon's own description says it may train models. Treat it as training-adjacent. Cloudflare's managed robots.txt, which Bot Preference Sync generates when a zone sets an AI policy to Block or, for Training, to Disallow, disallows Amazonbot alongside GPTBot, ClaudeBot, Google-Extended and meta-externalagent. It does not mention Meta-WebIndexer or Amzn-SearchBot, so those two stay allowed under that file.

How each one treats robots.txt

Amazon adds two things worth knowing. Its bots read page-level robots meta tags: “noarchive (do not use the page for model training), noindex (do not index the page) and none”. And they treat each host separately, so a subdomain needs its own robots.txt. Amazon's 30-day cache sits awkwardly with RFC 9309, which says crawlers “SHOULD NOT use the cached version for more than 24 hours”, but Amazon's same page says settings take about 24 hours to take effect. Plan for a day, and don't be surprised by a straggler.

What 39 large sites did with these bots (13 October 2026)

We fetched robots.txt from 40 well-known sites on 13 October 2026 and parsed each group with exact, case-insensitive token matching. One, stackoverflow.com, answered our client with HTTP 418 and a deny-all file whatever user agent we sent, so we left it out. A “full block” below means the bot's group contains Disallow: / without Allow: /. The sample leans heavily towards large publishers and platforms, so read it as a snapshot of how big sites behave, not as a rate for the web.

The sites were amazon.com, apple.com, bbc.co.uk, bbc.com, bloomberg.com, cloudflare.com, cnet.com, edition.cnn.com, ebay.com, economist.com, en.wikipedia.org, espn.com, etsy.com, facebook.com, forbes.com, ft.com, github.com, healthline.com, imdb.com, indeed.com, linkedin.com, medium.com, microsoft.com, npr.org, nytimes.com, quora.com, reddit.com, reuters.com, shopify.com, theguardian.com, theverge.com, tripadvisor.com, usatoday.com, walmart.com, washingtonpost.com, wikihow.com, wired.com, wsj.com and yelp.com.

Three patterns stand out. First, sites name Meta's training bot far more often than its search bot: 21 against 8. Many will have blocked meta-externalagent and assumed that covered Meta, which it doesn't. Second, eight of the 39 (facebook.com, imdb.com, linkedin.com, quora.com, reddit.com, reuters.com, wsj.com and yelp.com) start from User-agent: * with Disallow: / and allow only the bots they list. On those sites an unnamed bot is shut out, with one possible exception: Amazon says an unnamed Amzn-SearchBot follows the rules given to other search bots. The two sites that name Meta-WebIndexer without blocking it, reuters.com and indeed.com, both put it in an allow list; IMDb, which Amazon owns, allows Amazonbot, Amzn-SearchBot and Amzn-User. Third, Mistral's new search crawler, MistralAI-Index, appears in none of the 39 files, while its user fetcher is already blocked by seven.

# Excerpt, reuters.com/robots.txt, 13 October 2026 (an allow list; further down, User-agent: * gets Disallow: / apart from /plus/)
User-agent: meta-externalads
User-agent: meta-webindexer
User-agent: MicrosoftPreview
...
User-agent: OAI-SearchBot
...
Disallow: /finance/stocks/option
Disallow: /search

# Excerpt, imdb.com/robots.txt, 13 October 2026
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: ClarityBot
User-agent: Citoid
Allow: /
Disallow: /register

What the firewall did with them (a simulation)

robots.txt is a request; a CDN rule is enforcement. We requested five pages with a browser string and with the Meta-WebIndexer, Amzn-SearchBot and Amazonbot strings on 13 October 2026 at about 16:27 UTC. These requests came from our own IP address, not from Meta's or Amazon's networks, so they show how each site treats the name, not how it treats the verified bot. The method is the one in can ChatGPT, Claude and Perplexity read your website.

The Guardian's robots.txt doesn't name Meta-WebIndexer, so the file allows it, yet its edge refused the string. Wikipedia's robots.txt names none of these bots, and its edge refused only the Meta string, with a message inviting the operator to ask for unblocking. IMDb served its full page to the Amazonbot string while challenging both our browser string and Amzn-SearchBot. In each case the file and the firewall disagree, which is why checking robots.txt alone gives false comfort. Because these were spoofed requests, the real crawlers may fare better (if the site verifies them by IP) or the same.

How to verify a request really came from Meta or Amazon

Amazon publishes an IP list for each bot, as JSON shown on a web page: Amazonbot, Amzn-SearchBot and Amzn-User. When we fetched the Amzn-SearchBot list on 13 October 2026 its creationTime was 2026-09-08 and it listed individual /32 addresses, so expect it to change and re-download it rather than copying it once.

Meta's crawler page publishes no IP list. For FacebookExternalHit it advises allow-listing “the user agent strings or the IP addresses (more secure)” without saying where the addresses are. A practical check, which is our suggestion rather than a Meta-documented method, is to confirm that the source IP is announced by Meta's network, AS32934, which RIPEstat lists as “FACEBOOK - Facebook, Inc.”.

# Which network announces this IP? Meta's crawlers should show ASN 32934
curl -s 'https://stat.ripe.net/data/network-info/data.json?resource=203.0.113.7'

# Confirm who holds an ASN
curl -s 'https://stat.ripe.net/data/as-overview/data.json?resource=AS32934'

# Amazon: compare a claimed Amzn-SearchBot IP with the published list (the page embeds JSON)
curl -s https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ | grep -c '"203.0.113.7/32"'

Replace 203.0.113.7 with an address from your logs. A request with a Meta or Amazon token from an unrelated hosting network is an impostor, and you can block it without affecting the real crawler.

Allow, block or rate-limit?

Amazon ties one incentive to Amazonbot: “If you allow Amazonbot on your robots.txt, you may be eligible for benefits with Amazon Content Partners. Amazon Content Partners gives independent content creators a +1% affiliate commission boost on eligible sales, free hosting credits, and AI traffic management tools — all in one program.” That matters to affiliate publishers and to almost nobody else.

Crawl load is the most common reason people block these bots, and it is a real concern. In Cloudflare's May 2025 data, Meta-ExternalAgent made “a strong new entry at 19%” of AI-crawler traffic, behind GPTBot at 30% and ClaudeBot at 21%, with Amazonbot at 11%. But a block is a blunt answer to a volume problem. Amazon says its bots ignore crawl-delay, Meta's page doesn't mention it, and neither documents how its crawlers react to HTTP 429. A rate-limit rule at your CDN or server enforces a ceiling whatever the bot does, and keeps you in the index. If you are on Cloudflare, check what AI Crawl Control is doing first; our Cloudflare guide covers the 2026 settings. Note that Cloudflare's AI Crawl Control bot reference (last updated 23 April 2026) lists Meta-ExternalAgent, Meta-ExternalFetcher and Amazonbot but not Meta-WebIndexer or Amzn-SearchBot.

# Cited by Meta AI and Alexa, no training use by Meta or Amazon
User-agent: meta-webindexer
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

If your file starts from User-agent: * with Disallow: / and an allow list, add meta-webindexer and Amzn-SearchBot to the list explicitly. Amazon's fallback may let an unnamed Amzn-SearchBot follow your rules for other search bots, but naming it removes the guesswork, and Meta documents no fallback at all. If Cloudflare manages part of the file, the managed block is prepended to yours, so check the live robots.txt after saving. Its Content-signal line is explained in Content-Signal in robots.txt.

Mistral's three bots, briefly

Mistral now documents three crawlers with the same split. MistralAI-Index is “for automated crawling of the web for indexing purposes only” and its content “is not used for generative AI training of any kind”. MistralAI-User handles user actions in Vibe, Mistral's assistant, and “is not used for crawling the web in any automatic fashion”. MistralAI-Training “crawls web content to help build datasets for training Mistral generative AI models”. All three use the form Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots), with the token changed. As our sample shows, almost nobody names the Index and Training tokens yet, so most sites leave both under their * group.

Check your own site

Paste a URL into the free AI crawler checker. It reports a robots.txt verdict for Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAgent, Amzn-SearchBot, Amazonbot and the other major AI and search crawlers, using exact token matching, and tests whether your firewall turns AI search crawlers away. For the full picture, including whether your content is in the raw HTML and whether snippet controls hide it, run a free audit.

Being reachable doesn't guarantee a citation in Meta AI or Alexa; nobody outside those companies knows how their answers pick sources, and neither publishes a ranking guide. But a bot you block by accident can't cite you at all, and on the evidence of 39 large sites, the search bots are the ones most often forgotten.