llms.txt vs robots.txt vs Bot Protection for AI Access

Only one of these three actually stops an AI crawler: bot protection at the server or firewall level. robots.txt is a request that well-behaved bots choose to honour, and llms.txt is a menu you offer to models — neither blocks anything on its own. If you want real control over how AI reads your site, you need to know which layer does what and stack them deliberately.

The confusion is understandable. Three files-and-settings that all sound like they govern crawlers, and each one gets described online as if it were the answer. It isn't. Each solves a different problem, and using the wrong one leaves you thinking you're protected when you're wide open.

Here's the plain version, plus exactly how to set each up depending on whether you're on shared hosting or a VPS.

What each mechanism actually does

robots.txt asks, llms.txt guides, and bot protection is the only one that enforces — that's the whole distinction in one line.

robots.txt is a decades-old convention. You list crawlers and the paths you'd rather they skip. Compliant bots read it and obey; anything malicious ignores it entirely. It's a "please don't" note taped to the door, not a lock.

llms.txt is newer and does something almost opposite. Rather than keeping crawlers out, it's a curated Markdown file at your site root that helps language models find your best content — a clean map to your key pages, so a model summarising your site quotes the right things instead of guessing. It grants access thoughtfully; it never restricts it.

Bot protection is the only enforcement layer. This is a web application firewall, rate limiting, or server rules that inspect requests and block or challenge them before your site responds. It doesn't ask politely — it drops the connection.

What AI crawlers actually obey

The major AI companies publish named crawlers that respect robots.txt — but respect is voluntary, and plenty of scrapers ignore the file completely.

If your goal is to keep your content out of AI training sets, robots.txt is the honest first step because the big players do read it. OpenAI uses GPTBot, Anthropic uses ClaudeBot and anthropic-ai, Google's training crawler is Google-Extended, and Common Crawl — which feeds many datasets — uses CCBot. Naming these in robots.txt genuinely reduces your exposure.

But two gaps remain. First, retrieval is not training. When someone asks ChatGPT or Perplexity a live question, a separate user-triggered fetcher may pull your page in real time, and those often don't check the training-bot rules. Second, a huge amount of scraping comes from bots that never identify themselves and never read robots.txt. For those, the file is invisible.

Here's a robots.txt block that covers the main declared AI crawlers:

• User-agent: GPTBot — Disallow: /
• User-agent: ClaudeBot — Disallow: /
• User-agent: Google-Extended — Disallow: /
• User-agent: CCBot — Disallow: /
• User-agent: PerplexityBot — Disallow: /

Add each as its own User-agent line followed by Disallow: / in the file at yourdomain.com/robots.txt. Note that blocking Google-Extended does not affect normal Google search indexing — that's a separate crawler. One habit worth keeping: check your access logs a week after adding these lines. If you still see GPTBot hitting real pages with a 200 response, either your robots.txt isn't where the bot expects it or the request is coming from something spoofing the name — and a spoofed agent is exactly what robots.txt can never touch.

When llms.txt is worth your time

llms.txt is worth adding if you want AI answers to represent your site accurately — it's a publishing move, not a defensive one.

Think of a freelancer or small shop that keeps getting summarised badly in AI answers because a model latched onto an old landing page or a thin blog post. An llms.txt file lets you point models at the pages that carry your real story: services, pricing, the explainer articles you're proud of. It's a plain Markdown file with an H1 for your site name, a short blurb, then linked sections.

Here's a minimal one that lives at yourdomain.com/llms.txt:

• # Acme Studio
• > Independent web design studio building fast, no-nonsense sites for small businesses.
• ## Key Pages
• - [Services](https://acme.studio/services): what we build and what it costs
• - [Pricing](https://acme.studio/pricing): flat project rates, no surprises
• - [Guides](https://acme.studio/guides): our best how-to articles

Keep it short and current — a stale map is worse than none. Be realistic about adoption. Support for llms.txt is still patchy across models, so treat it as low-effort upside rather than a guaranteed channel. If you're spending an afternoon fighting AI crawlers, spend it on robots.txt and bot protection first, and add llms.txt when those are sorted.

How to combine all three on shared hosting

On shared hosting, lean on robots.txt plus whatever bot filtering your host provides, because you can't install a firewall of your own.

You don't get root access on shared plans, so raw firewall rules and custom rate limiting are off the table. What you can do still covers most of the risk:

• Write a real robots.txt naming the AI crawlers above — this handles the declared, well-behaved ones.
• Add rules in .htaccess to block specific user agents at the Apache level. Unlike robots.txt, this actually refuses the request rather than asking nicely.
• Use a security plugin if you're on WordPress — Wordfence or similar can rate-limit and challenge aggressive crawlers.
• Lean on your host's edge filtering. Good shared hosting already screens a layer of junk traffic before it reaches you.

The .htaccess block is the underrated one here. Drop this at the top of the .htaccess in your web root and it returns a 403 to every listed agent:

• RewriteEngine On
• RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
• RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
• RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
• RewriteRule .* - [F,L]

The pattern is worth understanding rather than just pasting: each RewriteCond is one agent, [NC] makes the match case-insensitive, and [NC,OR] chains it to the next line so any one match is enough. The last condition drops the OR — that's the line people get wrong. Then RewriteRule .* - [F,L] forbids the request. It only works against bots that send an honest user-agent string, but it's a genuine step up from robots.txt alone.

On TPC Hosting shared plans, our support engineers are on hand 24/7 to help you get an .htaccess rule right the first time instead of guessing — a misplaced rule can lock out legitimate visitors, so it's worth a second pair of eyes.

How to combine all three on a VPS

On a VPS you get the enforcement layer shared hosting can't offer: real firewall rules, rate limiting and IP-level blocking you control directly.

With root access, robots.txt and llms.txt still do their jobs, but now they sit on top of defences that don't depend on a bot's goodwill. This is where you can actually stop anonymous scrapers.

• Rate-limit at the web server. In Nginx, limit_req_zone caps how fast any single IP can hit you — brutal on scrapers, invisible to humans.
• Block by user agent in Nginx or Apache config for the declared AI bots, returning 403 or 444.
• Add fail2ban to auto-ban IPs that trip your rules repeatedly, so one aggressive scraper doesn't keep hammering you.
• Put a WAF or a CDN with bot management in front — this catches distributed scraping that no single-server rule can.

The Nginx rate limit is two lines in two places. Define the zone once in the http block:

• limit_req_zone $binary_remote_addr zone=crawlers:10m rate=10r/s;

Then apply it inside the server or location block you want to protect:

• location / {
•   limit_req zone=crawlers burst=20 nodelay;
•   # your existing proxy_pass or root config
• }

The 10m zone tracks roughly 160,000 IPs, rate=10r/s is the sustained ceiling per IP, and burst=20 absorbs short spikes before it starts returning 503s. Reload with nginx -t && systemctl reload nginx so a typo can't take the site down. Tune the rate to your real traffic — a busy shop with logged-in sessions needs more headroom than a static brochure site.

The stacking order matters: CDN/WAF first, then server-level rate limiting and user-agent blocks, then robots.txt and llms.txt for the polite crawlers. Each layer catches what the one above missed.

If you're weighing a move to a VPS specifically to get this control, TPC Hosting includes free migration, so you can bring your site over and build the firewall rules without downtime — and you've got 30 days to back out if it isn't the right fit.

The decision matrix in one glance

Pick the mechanism by your actual goal, not by which file sounds most authoritative.

The mistake to avoid is treating any one of these as complete. robots.txt without enforcement lets bad actors walk in. Bot protection without robots.txt still hands your content to compliant AI crawlers you'd rather opt out of. And llms.txt does nothing for privacy at all — it's the opposite job. Use them together, matched to what you're actually trying to achieve.

FAQ