Stay in Search, Say No to AI Training: The New Split

You can now tell crawlers "index me for search, but don't train your AI on my content" — and for the first time the big players have agreed on a shared way to say it. Cloudflare, working with Apple, Google and Microsoft, has proposed a system that separates the crawler's identity from its purpose, so search indexing and AI training stop being an all-or-nothing bundle.

Here's the short version: you add a couple of directives to robots.txt and, ideally, an HTTP header, declaring that search bots are welcome but AI-training bots are not. Bots that adopt the new "Accountable" designation commit to honouring that split and to identifying themselves honestly.

The catch — and it's a real one — is that this only binds the bots that choose to play along. Below is how to set it up today, what the label actually promises, and the gaps it doesn't close.

What the "Accountable" designation actually covers

"Accountable" is a voluntary pledge that a crawler will declare its identity, state its purpose, and respect purpose-specific rules — it is not a law or a technical lock. The point is to fix a long-standing mess: for years a single bot from a big company might crawl your pages for search results and quietly feed the same content into a training corpus, with no way for you to allow one and refuse the other.

The new framework asks crawlers to separate those jobs. An accountable crawler tells you which of three buckets it falls into:

• Search — building an index so people can find and click through to your site.
• AI input — fetching a page live to answer a user's question right now (retrieval, the "read it and summarise" case).
• AI training — collecting your content to bake into a model's weights.

The distinction that matters most is that middle one. Blocking training used to risk blocking the retrieval that sends AI-assistant users to your site. The accountable model lets you keep the traffic-generating behaviour and refuse the training that gives nothing back. It also commits participating companies to using verifiable bot identities, so you can tell a genuine Googlebot from something wearing its name.

How to configure the split, step by step

Set it up in two layers: a plain-English robots.txt that names purposes, plus an HTTP response header that repeats the rule for crawlers that read headers instead of files. Do both — different bots check different places.

Start with robots.txt in your web root. Allow the search bots you want, then disallow the training agents by name:

• Keep search: leave Googlebot, Bingbot and Applebot with full access.
• Block training: add explicit disallow blocks for the known training agents — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider.

A working example:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Note the pattern: Google-Extended and Applebot-Extended are the training identities, kept deliberately separate from Googlebot and Applebot so you can refuse one without losing the other. That separation is the whole design.

Then add a response header so the rule travels with the page. In Apache, drop this in .htaccess:

Header set X-Robots-Tag "noai, noimageai"

In Nginx, inside your server block:

add_header X-Robots-Tag "noai, noimageai";

The emerging convention uses noai for text and noimageai for images. Treat it as a signal, not a fence — it's read by the crawlers that have agreed to read it.

Where this still leaks

None of this stops a crawler that ignores the rules, and plenty do. robots.txt and response headers are requests, not enforcement. A scraper that doesn't identify itself, or lies about who it is, will walk straight past every directive above and take your content anyway.

The other leaks are quieter:

• Third-party republishing. If your posts get syndicated, quoted or mirrored elsewhere, a training bot can harvest that copy without ever touching your server or seeing your rules.
• Datasets already collected. Blocking a bot today does nothing about content vacuumed up last year. There is no recall button.
• Retrieval vs training blur. A single company may run one agent that both answers live questions and quietly logs content. The accountable model asks them to keep those separate; it can't force them to.
• Spoofed identities. Anyone can put Googlebot in a user-agent string. Verification (reverse DNS on the requesting IP) is the only reliable check, and most site owners never run it.

So set your expectations honestly: the split-access controls raise the cost and remove the excuses for the companies that want to be seen behaving well. They do nothing to the ones that don't care. That's still worth having — a clear, documented "no" matters if this ever ends up in front of a regulator or a court — but it is a signal, not a shield.

Backing the signal with an actual block

If you want teeth, enforce refusal at the server or firewall instead of only asking politely in a text file. This is where a signal becomes a block. Two practical routes:

Verify then allow. For the search bots you do want, confirm they're real before trusting the user-agent. A claimed Googlebot should reverse-resolve to a googlebot.com or google.com host; run host on the requesting IP and forward-confirm it. Anything failing that check gets treated as an impostor.

Block by user-agent at the edge. A firewall or WAF rule that returns 403 to known training agents is more reliable than robots.txt because it refuses the request rather than trusting the bot to refuse itself. In Nginx:

if ($http_user_agent ~* "GPTBot|ClaudeBot|CCBot|Bytespider") { return 403; }

If your host runs a managed firewall, ask them to apply this at the network layer so it never reaches your app. On TPC Hosting, our EU-based servers include Cloudflare integration, so you can turn on bot-management rules without editing config by hand — and if you're not sure which agents to block, the AI on our chat can point you at the current list any hour of the day, with a real engineer following up during business hours.

A quick reference for the common bots

Here's the mapping between each company's search identity and its training identity, so you know exactly what to allow and what to disallow.

Two things to keep straight. OpenAI splits cleanly: OAI-SearchBot for its search feature, GPTBot for training and general retrieval — allow the first, block the second if you don't want your content in answers. And there's no separate search-only identity for Anthropic or Common Crawl, so a disallow there is simply a full "stay out."

Should you actually block training?

Block training if your content is the product — original writing, tutorials, research, anything people would otherwise pay or visit for. Allow it if broad discovery matters more to you than protecting the words themselves.

For most small businesses and freelancers the honest answer is: block training, keep search. You want to be found, you want the click, and you gain nothing from a model reproducing your advice without sending anyone your way. A local plumber's service pages lose little to training; a copywriter's portfolio or a niche blog loses a lot.

Whatever you decide, decide it deliberately and write it down in your files. Review the agent list every few months — new bots appear constantly, and a disallow only works against names you've actually listed. If you'd rather not track that yourself, hosting that manages the block for you (and lets you back out within 30 days if it's not for you) takes the maintenance off your plate.

FAQ