Stop ChatGPT's Fetch Bot: What Actually Blocks It

If you added OpenAI's crawler to robots.txt and assumed ChatGPT could no longer read your pages, that assumption is now wrong. OpenAI's ChatGPT-User fetch bot — the one that pulls a live page when a user shares a link or asks about a URL — can reach content you disallowed, and OpenAI's own documentation explains why: it treats user-triggered fetches as an action a person requested, not automated crawling.

The short version: robots.txt is a polite request that well-behaved crawlers honour. A fetch triggered by a live user is a different category, and it does not reliably obey your rules. If you actually need to stop or shape what ChatGPT reads, you have to enforce it at a layer the bot cannot ignore — your web server, a WAF, or authentication.

Below is what works, what doesn't, and how to do it without accidentally locking out Googlebot and tanking your search traffic.

Why robots.txt no longer stops the ChatGPT fetch bot

robots.txt is voluntary, and OpenAI now classifies its live page fetch as a user action rather than a crawl — so the disallow you wrote may simply not apply.

There are three separate OpenAI agents, and they behave differently. Confusing them is why so many people think they're blocked when they aren't:

So blocking GPTBot in robots.txt keeps your content out of training data. It does nothing to stop a person pasting your URL into ChatGPT and getting the page summarised back to them, because that request comes through ChatGPT-User. If your goal is the second thing, robots.txt is theatre. You need enforcement.

Block at the server: the rule the bot can't argue with

A server-level block returns a 403 to the bot regardless of what it thinks the rules are — this is the control that actually holds.

The cleanest approach is matching the user-agent string and refusing the request. On Apache, drop these three lines into your .htaccess or vhost config:

• RewriteEngine On
• RewriteCond %{HTTP_USER_AGENT} "ChatGPT-User|GPTBot|OAI-SearchBot" [NC]
• RewriteRule .* - [F,L]

On Nginx, the equivalent is a single if — but it must sit inside a server { } or location { } block, not at the top level of the config, or Nginx will refuse to start:

• if ($http_user_agent ~* "ChatGPT-User|GPTBot|OAI-SearchBot") { return 403; }

User-agent matching is easy to set up and honestly good enough for most small sites, because OpenAI's bots identify themselves truthfully. The weakness is that a user agent is just a header — anyone determined can fake it. If you want a block that can't be spoofed, verify by the request's source instead. OpenAI publishes the IP ranges its bots use as JSON — openai.com/chatgpt-user.json for ChatGPT-User, plus openai.com/gptbot.json and openai.com/searchbot.json for the others. Pull those CIDR blocks and allow or deny on them, so a forged user-agent header gets you nowhere. That's more maintenance, but it's the version that actually stands up.

If you're on shared or managed hosting with us and you're not comfortable editing server config directly, this is exactly the kind of thing our support engineers set up for you — real people, any hour, no ticket-queue limbo.

Use a WAF to shape access, not just slam the door

A WAF lets you do more than block — you can rate-limit AI fetchers, allow them on some paths and deny them on others, and log everything so you know who's actually reading your site.

Cloudflare added a one-click AI bot block, and it's worth turning on if that's your stack. But the more interesting move is selective control. You rarely want to block all AI access — you might be happy for ChatGPT to summarise your public blog while keeping it away from your pricing pages, gated resources, or a members area. A WAF rule expressed as "if user-agent contains ChatGPT-User AND URI path starts with /account then block" gives you that precision without a blunt sitewide ban.

Three WAF patterns worth setting up:

• Path-based rules — allow AI fetchers on marketing pages, deny on /app, /account, /checkout.
• Rate limiting — cap requests per minute from known AI user agents so a burst of fetches doesn't spike your load.
• Logging and alerts — keep a record of AI-agent hits so you can see what's being read and react if behaviour changes.

A WAF is also the only one of these controls that catches spoofed agents cheaply, because you can combine user-agent matching with IP reputation and challenge suspicious traffic instead of trusting the header outright.

Authentication: the only guarantee

If content sits behind a login, no bot reads it — this is the one control with no asterisk.

Everything above is a fence a bot chooses to respect or an obstacle it has to route around. A login is a wall. If a page truly must never be read by ChatGPT, Perplexity, or anything else, the honest answer is: put it behind authentication. This is why bots can summarise a public news article but not your Gmail.

For content that's public but sensitive to scraping — think proprietary research, a paid knowledge base, or original data you don't want regurgitated — consider a soft gate: an email wall, a simple HTTP auth prompt, or requiring a free account. It costs you some reach, and that's the trade you're making. Decide deliberately rather than assuming a robots.txt line is protecting anything.

Don't accidentally block Google while you're at it

The single most common self-inflicted wound here is a broad bot rule that also catches Googlebot, Bingbot, or your uptime monitor — kill your search traffic and no AI policy matters.

Be surgical. Match specific AI user-agent strings, never a catch-all like "bot" — that word appears in Googlebot, Bingbot, DuckDuckBot and dozens of tools you want. After you deploy any block, verify it in both directions:

• Confirm the block works: curl -A "ChatGPT-User" https://yourdomain.com/ — you should get a 403.
• Confirm Google still gets through: curl -A "Googlebot" https://yourdomain.com/ — you should get a 200.
• Check for real: use Google Search Console's URL Inspection → Live Test. If Google can render the page, you didn't break search.
• Watch your logs for a day after any change and grep for 403s against known-good crawlers.

One more trap: don't put your AI blocks in robots.txt and stop there, then also add a server block, then forget which does what six months later. Keep robots.txt for the bots that honour it (GPTBot, OAI-SearchBot), and keep enforcement in one place — the server or the WAF — and leave a comment saying why, so the next person editing it isn't guessing.

A quick decision checklist

Match the control to what you're actually trying to achieve — the layers stack, they don't compete.

• Keep my content out of AI training data → block GPTBot in robots.txt.
• Stop live ChatGPT fetches of specific pages → server 403 or WAF rule on ChatGPT-User.
• Shape rather than ban (allow blog, block account area) → path-based WAF rules.
• Stop spoofed agents pretending to be a browser → verify by OpenAI's published IP ranges.
• Guarantee nothing reads it, ever → put it behind a login.

The reporting that kicked this off is a useful reminder: the AI web moves faster than the standards meant to govern it, and "I added a line to robots.txt" is not a policy. Pick the layer that matches your risk, verify it with curl, and move on.

FAQ