Stop ChatGPT's Fetch Bot: robots.txt Won't Cut It

If you added Disallow lines to block ChatGPT and expected the traffic to stop, it won't. OpenAI now documents that its live page-fetching bot — the one that grabs a URL when a user asks ChatGPT about it — treats robots.txt as optional, because it's acting on behalf of a person, not crawling on its own initiative. So the only things that reliably keep that bot off a page are controls at the server or network layer: firewall rules, WAF filters, and authentication.

The short version: robots.txt is a request, not a lock. Well-behaved training crawlers honour it. A user-triggered fetch treats itself more like a browser, and browsers don't ask permission. If you actually need a page kept out of ChatGPT, you have to enforce it.

Below is what works, what doesn't, and how to do it without accidentally slamming the door on Googlebot and tanking your search traffic.

Why robots.txt doesn't stop ChatGPT's fetch bot

robots.txt is a voluntary standard, and OpenAI classifies its live fetch as user-initiated action rather than crawling — so it deliberately doesn't apply the file's rules to it.

OpenAI runs more than one agent, and this is where people get confused. GPTBot is the training crawler and it does respect robots.txt. OAI-SearchBot powers the search feature and also reads the file. But ChatGPT-User — the agent that fetches a URL because someone pasted it or asked a question about it — is documented to potentially ignore robots.txt, on the logic that a human asked for that specific page in real time.

Whether you agree with that reasoning or not, the practical takeaway is the same. A directive that depends on the other side choosing to obey is not a control. If the content matters, you enforce it below the polite-request layer.

What actually blocks an AI fetch bot

Three things reliably stop a fetch bot: authentication, a server-level deny rule, and a WAF filter matching the bot's user agent or IP. Everything else is a suggestion.

Ranked by how airtight they are:

• Authentication (strongest). A login wall, HTTP basic auth, or a signed URL. No token, no page. A fetch bot can't guess a password, so this is the only method that's truly content-proof.
• Server-level deny by user agent. Apache or Nginx returns a 403 before the page renders. Effective as long as the bot sends an honest user-agent string — and OpenAI's do.
• WAF / edge rules. A rule at Cloudflare, your firewall, or your host's edge that matches the user agent or known IP ranges and blocks or challenges the request.
• robots.txt (weakest). Fine for the training crawler. Ignored by the live fetch. Keep it, but don't rely on it alone.

The distinction that matters: robots.txt asks the visitor to leave; a server block never lets them in the door. For anything you genuinely don't want summarised in a chat window, use the door.

Block ChatGPT-User at the server without touching Google

Match on the exact user-agent strings OpenAI publishes and return a 403 — that blocks the AI bots while leaving Googlebot, Bingbot and human visitors completely untouched.

The current OpenAI agents you'd target are GPTBot, ChatGPT-User and OAI-SearchBot. On Apache, in .htaccess or a vhost block:

• RewriteEngine On
• RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot) [NC]
• RewriteRule .* - [F,L]

On Nginx, inside the server block:

• if ($http_user_agent ~* "(GPTBot|ChatGPT-User|OAI-SearchBot)") { return 403; }

Because you're matching named agents, Google's crawlers never hit these rules — their user agents don't contain those strings. That's the whole point: you're being surgical, not swinging a bat. Test it before you trust it. Run curl -A "Mozilla/5.0 (compatible; ChatGPT-User/1.0)" https://yoursite.com/page and confirm you get a 403, then run curl with a normal user agent and confirm you still get 200.

One caveat: user-agent matching only works while the bot identifies itself honestly. OpenAI's do today. If you want a control that doesn't depend on that, add IP-range blocking — OpenAI publishes the address ranges for each agent, and you can deny those at the firewall so a spoofed user-agent string gets you nowhere.

The mistakes that accidentally block Google too

The two ways people cut off Google while trying to block AI are over-broad user-agent rules and blanket "bot" filters at the WAF — both catch far more than they should.

Watch for these:

• Matching on "bot" alone. A rule that blocks any user agent containing "bot" will catch Googlebot, Bingbot and half the monitoring tools you rely on. Always match the full, specific agent name.
• A WAF "block AI scrapers" toggle you don't read. Some managed rulesets bundle search engines into their AI category. Check the exact list before flipping it on, then verify in Search Console that Google can still fetch.
• Blocking IP ranges you didn't verify. Copy-pasting an old block list can catch shared ranges. Use the ranges OpenAI publishes, not a random gist.
• A noindex meta tag as your "AI block." noindex tells search engines not to list the page — it does nothing to stop a live fetch, and it can quietly remove you from Google.

After any change, use the URL Inspection tool in Google Search Console to fetch a blocked page as Google. If Google renders it fine and your curl test with the ChatGPT user agent returns 403, you've got it right.

What to block, and what to leave open

Block your fetch bots from private, gated, or commercially sensitive content — and think hard before blocking your public marketing pages, because AI answers are becoming a real referral source.

A quick way to decide:

There's a real trade-off here. Block ChatGPT-User from your blog and you also disappear from the answers ChatGPT gives when someone asks about your topic — increasingly a source of qualified visitors. Blocking makes sense for content you sell or content that's genuinely yours to protect. For pages whose job is to be found, keeping them fetchable is the smarter call.

If you'd rather not hand-edit vhost files, this is the kind of thing our support team does daily. TPC Hosting runs on EU servers with real engineers on hand 24/7, so you can send us the exact behaviour you want — block ChatGPT-User, keep Google — and we'll put the rules in place and test them with you.

A 10-minute setup you can do today

Add the server-level user-agent block, lock down staging with basic auth, verify with curl, then confirm Google still fetches in Search Console — that's the whole job.

The checklist:

• Decide which pages actually need protecting (use the table above).
• Add the Apache or Nginx rule matching GPTBot, ChatGPT-User and OAI-SearchBot.
• Put HTTP basic auth on any staging or dev subdomain.
• Optionally add OpenAI's published IP ranges to a firewall deny list for spoof resistance.
• Run curl with a ChatGPT user agent — expect a 403.
• Run URL Inspection in Search Console — expect a clean fetch.

Keep your robots.txt entries too. They still do the right thing for GPTBot's training crawl, and there's no harm in the polite request — just don't mistake it for the lock.

If any of this feels fiddly, that's fair — bot management sits in an awkward spot between SEO, security and server config. TPC Hosting includes free migration and gives you 30 days to back out, so you can move in, have our engineers set the rules up correctly, and see the results before you commit.

FAQ