Who Blocks Meta's AI Crawlers? 99 Big Sites Shut Out OpenAI's and Left Meta's Open

Among the 2,976 live sites in the world's top 5,000 whose robots.txt we could read on 13 October 2026, 9.8% block Meta-ExternalAgent, Meta's crawler for training foundation models and indexing content for its products. That's fewer than block OpenAI's GPTBot (11.6%) or Anthropic's ClaudeBot (10.6%). And 111 sites blocked GPTBot while leaving Meta-ExternalAgent open. We re-read their robots.txt on 17 October: 99 of the 107 we could still fetch hadn't changed, among them CNN, ESPN, TechCrunch, CBS News, the LA Times and Harvard Business Review.

Why this matters this month

Meta's updated Commercial Terms take effect on 30 October 2026. According to Unite.AI and Nieman Lab, a version of the terms let Meta crawl websites linked to business accounts on Facebook, Instagram and WhatsApp and use the content to “provide, develop, improve, research, train”. On 16 October the News/Media Alliance's chief executive, Danielle Coffey, called the terms “outrageous and probably illegal”. Meta then removed the training wording, and its spokesperson Andy Stone said “content owners can use robots.txt flags to indicate their preferences”, adding that “there was no change to how Meta uses web crawlers”.

So, by Meta's own account, robots.txt is the lever. We already had every top-5,000 site's robots.txt from our October crawler study, parsed for 31 AI crawlers including all four of Meta's.

Meta's four crawlers, and what each one does

Most of those blocks are deliberate: 211 sites name Meta-ExternalAgent in their own group. The rest fall under a rule for all crawlers. Meta-ExternalAds is almost never named (3 sites), so its 3.1% comes from sites that block every unnamed crawler. Of the 291 sites that block Meta-ExternalAgent, 144 still allow Meta-WebIndexer: they've opted out of training while staying citable in Meta AI.

The gap: OpenAI blocked, Meta not

A robots.txt group applies only to the crawler it names (RFC 9309). A site that writes `User-agent: GPTBot` and `Disallow: /` has said nothing to Meta-ExternalAgent, which falls back to the rule for all crawlers, usually “allow”. On 13 October, 111 live sites were in that position. On 17 October we fetched their robots.txt again and ran it through the same parser our audit uses: 107 answered, and 99 still blocked GPTBot while allowing Meta-ExternalAgent.

This doesn't mean any of these publishers wants Meta to train on its pages. Most AI blocklists were written in 2023 and 2024, when GPTBot, CCBot and Google-Extended were the names everyone copied. Meta-ExternalAgent arrived later and simply isn't in many of them. The reverse also happens: 57 sites block Meta-ExternalAgent but not GPTBot.

What to add to robots.txt

If you want out of Meta's model training but would still like Meta AI to be able to cite you, block Meta-ExternalAgent and leave Meta-WebIndexer alone:

# Meta: no training, keep Meta AI citations
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else, including Meta-WebIndexer
User-agent: *
Allow: /

Meta says crawlers may cache robots.txt for up to 24 hours, so allow a day for the change. To write a file covering every AI crawler at once, use the AI robots.txt generator; for exact user-agent strings to look for in your logs, see the AI crawler list. Then run a free audit to check that your CDN or firewall agrees with the file.