Your Site, Your Rules: Choosing Which AI Bots Can Read It

You can decide, right now, which AI bots are allowed to read your site — and treat them differently depending on what they want. The three groups worth separating are search crawlers (they help people find you), AI agents (they act on a user's behalf, like booking or buying), and training crawlers (they scrape text to train models). Most small sites should keep search open, decide agents case by case, and block or limit training.

This became a real choice for everyone recently. Cloudflare's Content Independence Day rollout now lets any site owner split those three bot classes instead of lumping all automated traffic together. That's the fresh part: you no longer have to choose between "let everything in" and "block the lot."

Here's how to think it through without a legal team or a week of reading, plus the exact files and settings to change today.

The three kinds of AI bot, and why they're not the same

Treat AI bots as three separate jobs — discovery, action, and training — because allowing one doesn't mean allowing the others. Bundling them together is the mistake that leads people to either block traffic they wanted or leak content they didn't.

Search crawlers index your pages so they show up in results and, increasingly, in AI-generated answers. Googlebot is the obvious one, but Google now also uses Google-Extended for its AI features, and OpenAI runs OAI-SearchBot for ChatGPT's search results. Blocking these usually means less visibility.

AI agents visit a specific page because a real person asked an assistant to do something — check your opening hours, compare a price, fill a form. OpenAI's ChatGPT-User and Anthropic's Claude-User fall here. This traffic is small today and growing, and it often represents a genuine visitor at the other end.

Training crawlers download your content in bulk to feed a model. GPTBot, ClaudeBot, CCBot (Common Crawl) and Google-Extended for training sit here. They give you nothing back directly — no click, no customer, no credit — which is why most small businesses draw the line here first.

What you actually gain and lose with each choice

The core trade-off is discoverability versus control: the more AI systems can read you, the more places you appear in answers — but the less say you have over how your words get reused. Weigh it per group rather than as one decision.

My position: keep search open, allow agents, and block training unless you have a reason to share. The reasoning is simple. Search still sends you customers. Agents usually represent a person trying to do business with you — blocking them is like ignoring the phone. Training gives you no click and no control, so the burden of proof sits on why you'd allow it.

There are exceptions. If your content is the product — a paid knowledge base, original research, a photography portfolio — you may want to block training and even limit agents to protect what you sell. And if you're chasing brand mentions and don't mind being quoted, letting training crawlers through is a defensible bet.

A quick decision framework for your site

Pick your stance by asking one question per bot class: does this traffic plausibly bring me a customer or protect my work? Run through this and you'll have your policy in five minutes.

• Do I want to appear in Google and AI search answers? Almost always yes → allow search crawlers.
• Would I answer a phone call from a customer's assistant? If yes → allow AI agents. If you sell gated content → limit them.
• Is my writing or media the thing people pay me for? If yes → block training crawlers. If you mainly want reach and mentions → allow them.
• Am I under strict client or GDPR obligations about content reuse? If yes → block training, document the decision.

Most freelancers and small shops land on the same result: search yes, agents yes, training no. That single line covers the majority of sites we host, and it's easy to change later if your view shifts.

How to set it up with robots.txt

The fastest way to signal your policy is a well-formed robots.txt in your site root, listing each bot by its user-agent and what it may crawl. It's a request, not a wall — reputable AI companies honour it, and it costs you nothing to add.

Here's a starter that keeps search open, welcomes agents, and turns away training crawlers:

• Allow normal search: leave Googlebot, Bingbot, OAI-SearchBot and Google-Extended unblocked.
• Block training: add User-agent: GPTBot / Disallow: /, then repeat for ClaudeBot, CCBot, anthropic-ai and Bytespider.
• Allow agents: don't disallow ChatGPT-User or Claude-User unless you're gating content.

Save it as plain text, upload it to your document root, and check yourdomain.com/robots.txt loads correctly. Give each bot its own User-agent block — one directive per bot reads more reliably than a clever combined rule.

One honest limitation: robots.txt is voluntary. Well-behaved crawlers respect it; a badly-behaved scraper may not. For content you truly can't afford to leak, you need enforcement, not a polite request.

When you need real enforcement, not just a request

If a bot ignores your rules, the only thing that stops it is a block at the server or network edge that returns a 403 before the content is served. This is where Cloudflare's new split matters: you can now toggle Search, Agent and Training bots separately in the dashboard and have the network enforce it, rather than trusting a text file.

You don't have to be on a big plan to do this. For hosting-level control you can also block by user-agent in your server config — an Nginx if ($http_user_agent ~* "GPTBot|ClaudeBot|CCBot") rule returning 444, or the Apache equivalent with mod_rewrite. That runs before your pages render, so it actually stops the download.

A sensible layered setup for a small business:

• robots.txt — states your policy for well-behaved bots.
• Server or CDN rules — enforce it against the ones that ignore robots.txt.
• Rate limiting — catches aggressive crawlers hammering your site regardless of who they claim to be.

If you host with TPC Hosting and want a hand adding user-agent blocks or rate limits, our engineers are on support 24 hours a day and will set the rules up with you — no ticket ping-pong. Since everything runs in the EU, your logs and content stay under GDPR rules, which matters if you're deciding what leaves your site in the first place.

Review it once a quarter

Set a calendar reminder to re-check your bot policy every three months, because the bot names and the companies behind them change faster than most other parts of your site. New crawlers appear, existing ones rename their user-agents, and your own priorities shift as AI search sends more or fewer visitors.

When you review, pull your access logs and grep for user-agents you don't recognise. If a new AI crawler is taking a real chunk of your bandwidth, decide which of the three buckets it belongs in and add a line for it. Ten minutes a quarter keeps you in control instead of reacting after the fact.

The point of all this isn't to fight AI — it's to make a deliberate choice instead of an accidental one. Your site, your rules.

FAQ