Content-Signal in robots.txt: search=yes, ai-train=no Explained

Content-Signal is a line in robots.txt that states how you want your content used after it has been crawled. Cloudflare introduced it in September 2025 and writes it into the robots.txt files it manages, usually as Content-Signal: search=yes, ai-train=no. That means yes to search indexes, no to AI training, and no stated preference on using your pages to ground AI answers. It is a preference, not a block.

Who reads it is the part most guides skip. On 13 October 2026, the crawler documentation of OpenAI, Anthropic and Perplexity did not mention it, and Google’s robots.txt parser ignores fields it does not support, which is why Search Console flags the line as “Syntax not understood”. That warning is harmless. Whether you add the line is a policy choice; it will not change whether AI systems cite you.

The exact syntax Cloudflare serves

This is the managed block Cloudflare documents for zones with Bot Preference Sync on (documentation updated 8 October 2026). Note the lowercase “signal” and the spaces after the commas:

# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

# (then the same Disallow group for Applebot-Extended, Bytespider, CCBot,
#  ClaudeBot, Google-Extended, GPTBot and meta-externalagent)

# END Cloudflare Managed Content

Above it, Cloudflare places a commented policy text that defines the terms. Robots.txt parsers ignore comments, so the definitions are for people and lawyers, not machines:

The policy text also declares, in capitals, that restrictions expressed this way are express reservations of rights under Article 4 of the EU’s 2019 copyright directive on the digital single market. That is Cloudflare’s legal position, written into your file; we are not lawyers, and whether it holds is for courts to decide.

What yes, no and a missing value mean

“Yes” means go, “no” means stop, and the absence of a signal conveys no meaning.

The missing value is the subtle one. Cloudflare’s managed file deliberately leaves out ai-input: “We will not serve an ‘ai-input’ signal for our managed robots.txt customers. We don’t know their preference with respect to that signal, and we don’t want to guess.” And because Cloudflare’s search excludes AI-generated summaries, search=yes on its own is not a yes to AI Overviews, ChatGPT answers or Perplexity. A site that wants to be quoted in AI answers and says only search=yes, ai-train=no has stated no preference on the use it cares about most.

What real robots.txt files say (13 October 2026)

On 13 October 2026 we requested /robots.txt from 81 well-known sites: large publishers, SaaS companies, retailers, website builders and SEO blogs. 74 returned a readable file. Eleven ordinary files contained a Content-Signal line. Four of them are Cloudflare’s own sites (www, blog and developers.cloudflare.com, and contentsignals.org), and a fifth is isitagentready.com, the readiness checker Cloudflare’s dashboard links to. A twelfth line came inside an HTTP 418 refusal to our client rather than an ordinary file. None of the 74 used the IETF’s Content-Usage. This is a convenience sample, not a census, but it shows the patterns. Verbatim excerpts:

# vercel.com/robots.txt (first lines)
User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

# webflow.com/robots.txt (first lines)
User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# developers.cloudflare.com/robots.txt (after the policy comments)
User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# www.patreon.com/robots.txt (excerpt)
User-agent: PetalBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: TikTokSpider
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

• The line has spread beyond Cloudflare’s managed file. vercel.com and zapier.com are served by Vercel, not Cloudflare, and still publish it.
• Cloudflare’s own sites opt in to everything, training included. The ai-train=no default goes only to customers who turned on managed robots.txt to refuse training, so it reflects their choice, not a recommendation for every site.
• Patreon puts the line inside the PetalBot and TikTokSpider groups. If a reader treats it like the other lines in a group, it applies only to the crawlers that group names, so that preference is addressed to those two crawlers alone.
• Spelling varies (Content-Signal, Content-signal) and so does the order of the values. A parser has to accept any capitalisation and any order.

Who actually reads Content-Signal

So the honest answer to “who reads it” is: no major AI crawler documents doing so, and the party that acts on it is Cloudflare, through its own network rather than through the line. When Training is set to Disallow, Cloudflare’s documentation says Bot Preference Sync “adds managed directives and enables edge blocking for AI training crawlers.” The edge block does the work. Cloudflare says as much itself:

It’s important to remember that content signals express preferences; they are not technical countermeasures against scraping. Some companies might simply ignore them.

If you need a crawler to stay out, name it in a Disallow group and, on Cloudflare, use the Training or Search policy. Our guide to Cloudflare’s AI Crawl Control settings explains which setting does what since 15 September 2026, and GPTBot vs OAI-SearchBot covers the user-agent rules OpenAI does document.

Content-Signal vs the IETF’s Content-Usage

The IETF’s AI Preferences working group (aipref) is standardising a related idea under a different name. Two working-group drafts matter: a vocabulary (draft-ietf-aipref-vocab-08, 14 September 2026) and a way to attach it to content (draft-ietf-aipref-attach-05, 19 August 2026, by Gary Illyes of Google and Martin Thomson of Mozilla). Both are aimed at the Standards Track and neither is an RFC. The vocabulary draft is blunt about its state: its contents “DO NOT REFLECT CONSENSUS of the Working Group either in whole or part.” Cloudflare’s own Internet-Draft for Content Signals, draft-romm-aipref-contentsignals-00, was last revised on 1 October 2025 and has expired.

The practical consequence: the two lines are not interchangeable. Under the draft’s processing rules, a value that is not the token y or n leaves the preference “unknown”, so a Content-Usage parser handed search=yes learns nothing. If you want to be read by tools that implement the IETF draft as well, you would need a second line. This is the draft’s own example:

User-Agent: *
Allow: /
Disallow: /never/
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

User-Agent: ExampleBot
Allow: /
Content-Usage: train-ai=y

No crawler we checked documents reading Content-Usage either. Adding it is cheap and forward-looking, with the same caveat as Content-Signal: it states a preference and enforces nothing.

The Search Console “Syntax not understood” warning

Search Console’s robots.txt report (under Settings) lists parsing issues in your file, and it flags Content-Signal because Google’s parser does not support the field. Cloudflare’s documentation addresses it directly:

Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard. However, we have observed no impact on crawling rates or SEO as a result of these reports.

Google’s own documentation agrees on the mechanics. The robots.txt report help says “Google ignores the lines with issues and uses those that it can parse” and that warnings “do not prevent a rule from being used.” The robots.txt standard, RFC 9309, says crawlers may interpret records outside the protocol but that “Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in Section 2.” Your User-agent, Allow and Disallow lines keep working. You can leave the warning alone.

What is worth checking is that the rest of the file is right, because that is what crawlers act on. Our AI crawler checker reads the file per bot, and Can ChatGPT, Claude and Perplexity read your website? walks through every access gate.

Should you add a Content-Signal line?

It is harmless to Google, cheap to add and may matter legally in the EU, but it is not enforcement and not a visibility lever. Decide what you want first:

• You want AI answers to use and cite you: if you add the line, include ai-input=yes, because a missing value says nothing. Then do what actually decides access: allow OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt and at your CDN.
• You want to refuse training: the line alone does nothing for crawlers that don’t read it. Add Disallow groups for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and the others, and on Cloudflare use Disallow AI Training.
• Avoid search=no unless you mean it. Google ignores the line today, but any crawler that honours it would read it as leaving search.
• If you use Cloudflare’s Markdown for Agents, check the header. When your origin sets no content-signal header, Cloudflare adds “a default Content-Signal: ai-train=yes, search=yes, ai-input=yes” to Markdown responses, which contradicts an ai-train=no line in robots.txt. Set the header at your origin to keep one policy.

For a business that wants to be found in AI answers but keep its pages out of training sets, this is a consistent file. Swap the training groups if you are happy to be trained on:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Training crawlers and control tokens: refused on purpose
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Because the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) are not named, they fall under User-Agent: * and are allowed. Disallowing Google-Extended and Applebot-Extended does not remove you from Google or Apple search; both companies say so in their crawler documentation.

What our audit reports

Our free audit reads Content-Signal and Content-Usage lines in robots.txt and the matching HTTP headers, and lists them under AI usage signals. It reports them and never scores them; the methodology explains why. If it sees Cloudflare plus ai-train=no, it prompts you to check your Training setting, because the setting behind the line, not the line itself, decides which crawlers get through. For the wider question of which AI tactics have evidence behind them, see GEO myths, graded.