Back to Article List

Opt Out of Google, Microsoft & OpenAI AI Training

Opt out din antrenarea AI: Google, Microsoft & OpenAI - Opt Out of Google, Microsoft & OpenAI AI Training

There's now a single control you can put in your robots.txt to tell Google, Microsoft and OpenAI not to use your pages for AI model training. Set the Content-Usage preference, or spell it out with a robots.txt rule, and all three vendors agree to read it the same way. The headline win: you no longer have to maintain a separate block for every crawler.

The catch is that opting out of training is not the same as opting out of search, and it's not the same as blocking the live crawlers that fetch your page to answer a question in real time. Those are three different things, and if you blanket-block everything you'll quietly vanish from places you actually want to be.

Here's exactly what to add, what each provider does with it, and how to refuse training without hurting your visibility.

The one line to add right now

The shared control came out of the IETF's AIPREF working group, and the preference it defines is ai-train. The cleanest way to express it is the HTTP response header on the pages you want excluded:

  • Content-Usage: ai-train=n

That single header tells participating AI vendors: don't use this page for training. If you'd rather carry the same instruction in robots.txt — handy when you want a sitewide default without touching headers — use the Content-Usage directive there instead:

  • User-agent: *
  • Content-Usage: ai-train=n

A quick accuracy note, because a wrong token silently does nothing: the preference name is ai-train and the opt-out value is n (for "no"). That's the string the spec defines — not a made-up bot name. The header and the robots.txt directive carry the same preference; pick whichever matches how granular you need to be.

The header wins when it's more specific. A sitewide directive is the blunt instrument; the header lets you say "train on these blog posts, but not on this one case study." If you only want to protect part of the site — say a members area or a paid archive — set the header on those routes and leave the rest open.

What each provider actually does with it

All three honour the opt-out for training, but they handle the edges differently — and that's the part worth understanding before you assume one line covers everything.

The short version: ai-train=n governs training only. It does not automatically stop a model from fetching your live page to answer a user's question right now (that's "grounding" or retrieval), and it has nothing to do with whether you appear in ordinary search results.

Use caseCovered by the training opt-out?
Model training on your contentYes — this is what it blocks
Live retrieval to answer a queryNo — controlled by separate crawler rules
Appearing in Google/Bing searchNo — governed by Googlebot / Bingbot
AI Overviews / Copilot answersPartly — depends on the crawler, not the preference

Google reads ai-train=n as a training signal and keeps it separate from Googlebot (search) and Google-Extended (its older AI-training token). Microsoft treats it as a training opt-out distinct from Bingbot. OpenAI maps it against its own crawlers — GPTBot (training), OAI-SearchBot (search indexing) and ChatGPT-User (live fetch when a user asks). Setting the preference stops the training behaviour; it does not touch the search or live-fetch bots.

So if your goal is "stay in search, stay quotable in real-time answers, but don't feed the training set," the preference alone does that cleanly. If your goal is "block everything AI, including live answers," you need the extra lines below.

How to refuse training but stay discoverable

To keep your pages in search while refusing training, block only the training agents and leave the search and retrieval crawlers alone. This is the setup most small-business owners actually want.

Here's a robots.txt block that opts out of training across the board while keeping you indexable in Google and Bing, and keeping you eligible to be cited in live AI answers:

  • User-agent: *
  • Content-Usage: ai-train=n
  • User-agent: Google-Extended
  • Disallow: /
  • User-agent: GPTBot
  • Disallow: /

Why keep both the ai-train preference and the older named bots? The shared preference is new, and not every tool or cache has adopted it yet. Listing Google-Extended and GPTBot explicitly is belt-and-braces until the standard is universally respected. Notice what's not in that list: Googlebot, Bingbot, OAI-SearchBot and ChatGPT-User. Leaving those open is deliberate — it's how you stay findable.

If you'd rather go all-in and block live AI retrieval too, add OAI-SearchBot and ChatGPT-User with Disallow: /. Just know the trade: you're giving up the chance to be the cited source when someone asks ChatGPT or Copilot a question your page could answer.

Where people get this wrong

The most common mistake is blocking your whole site with User-agent: * Disallow: / and assuming it only affects AI — it blocks Google search too, and your traffic will drop within weeks.

A few more traps worth avoiding:

  • Robots.txt is a request, not a wall. Compliant crawlers obey it; it does not stop scrapers that ignore the rules. For content you genuinely can't let out, put it behind a login, not just a directive.
  • One robots.txt per host. It must live at the domain root and apply to that host. A subdomain like shop.yourdomain.com needs its own file.
  • Case and spelling matter. Tokens are matched as written. A typo in ai-train or GPTBot silently does nothing.
  • The header and robots.txt can disagree. If you set Content-Usage: ai-train=n on a page but allow training sitewide, the more specific page-level header should win — but don't rely on contradictions. Keep them consistent.

One group that often slips through: the big dataset crawlers that aren't part of this agreement. CCBot (Common Crawl) is the obvious one — its archive feeds plenty of model training, and it does not read the ai-train preference. If you want it out, you block it by name with its own User-agent: CCBot / Disallow: /. And even then, the login-wall point stands: anything you truly can't afford to leak should sit behind authentication, because a non-compliant scraper won't ask permission at all.

After you edit, check it. Fetch yourdomain.com/robots.txt in a private browser window to confirm the live version updated, then open Google Search Console's robots.txt report. That report shows which file Google last fetched, when it fetched it, and whether it parsed cleanly — look for a green "Fetched" status and a timestamp that matches your edit, not an old cached copy. You can also paste a specific URL into the tester to confirm Googlebot still reads it as Allowed. If a path you meant to keep open comes back as blocked, you've caught the mistake before your rankings do.

A sensible default for small sites

For most freelancers and small businesses, the right call is: opt out of training, stay in search, and stay quotable in live answers. You lose nothing you were earning money from, and you stop your work being baked into a model for free.

The one group that might choose differently: publishers and anyone whose traffic depends on being the answer. If AI answers already siphon your clicks, blocking live retrieval as well can make sense — but test it, watch your referral data, and be ready to reverse it.

If your site is hosted with us at TPC Hosting, editing robots.txt is a two-minute job in the file manager, and because everything sits on EU servers under GDPR, controlling what leaves your site is the sort of thing we take seriously anyway. If you're not sure which crawlers to keep and which to drop, our AI on chat can talk it through at any hour, and real engineers follow up during business hours to check your file is doing what you meant.

FAQ

Does the AI training opt-out remove me from Google Search?

No. The ai-train preference only controls AI training, not search indexing. Googlebot and Bingbot are separate crawlers, so your pages stay in ordinary search results unless you block those agents specifically.

Do I still need Google-Extended and GPTBot if I use the ai-train preference?

For now, yes — keep them as a backup. The shared preference is new and not every system honours it yet, so listing the older named training agents alongside it makes sure you're covered until the standard is universally adopted.

Will robots.txt stop every AI company from training on my site?

No. Robots.txt is a voluntary request that compliant crawlers obey, but it cannot stop scrapers that ignore the rules — Common Crawl's CCBot, for instance, needs its own block and still won't bind a bad actor. For content you truly cannot let out, put it behind a login rather than relying on a Disallow line.

What's the difference between the robots.txt rule and the Content-Usage header?

Both carry the same ai-train=n preference. The robots.txt directive applies sitewide or to whole paths, while the Content-Usage header lets you opt out page by page. Use the header when you need finer control; the more specific signal should take precedence.

Can I opt out of training but still be cited in ChatGPT or Copilot answers?

Yes. Set the training opt-out and block the training agents (GPTBot, Google-Extended) but leave the live-fetch crawlers like ChatGPT-User and OAI-SearchBot open. That refuses training while keeping you eligible to be the cited source in real-time answers.