AI Crawlers Eating Your Bandwidth? How to Manage Bot Traffic

If your site feels sluggish and your bandwidth graph keeps creeping up without a matching jump in real visitors, AI crawlers are a likely culprit. Bots from OpenAI, Anthropic, Perplexity and a growing pile of scrapers now hit sites hard enough to show up as a server-load problem, not just an SEO footnote.

The fix is not to block everything. It's to read your logs, tell the useful bots from the greedy ones, and slow the greedy ones down while leaving Google and Bing alone. You can do all of it today with tools you already have — cPanel's raw access logs, a few lines in robots.txt, and some .htaccess or Nginx rules on a VPS.

Here's how to find them, sort them, and put a limit on the damage.

What's actually eating your bandwidth

Three different kinds of bots are hitting your site, and they want three different things — so you should treat them differently.

First, search crawlers like Googlebot and Bingbot index your pages so people can find you. You almost never want to block these. Second, AI training crawlers like GPTBot, ClaudeBot, CCBot (Common Crawl) and Google-Extended scrape your content to train models. They give you nothing back and can be aggressive. Third, AI agent fetchers like OAI-SearchBot, PerplexityBot and ChatGPT-User fetch a page live when someone asks a chatbot a question — these can drive a trickle of referral traffic, so blocking them is a judgement call.

The reason this shifted from an SEO concern to a hosting concern is volume and pattern. A training crawler doesn't respect your sitemap priorities. It requests everything — every tag archive, every paginated comment page, every faceted-search URL — often several times a day. On a shared plan that shows up as CPU throttling and slow page loads for real visitors. On a small VPS it shows up as a memory-hungry PHP-FPM pool getting hammered.

How to spot AI crawlers in your server logs

Open your raw access logs and filter by user-agent — that single field tells you who's knocking and how often.

In cPanel, go to Metrics → Raw Access and download the compressed log, or use Metrics → Errors for a quick look. On a VPS with SSH access, your logs live in /var/log/nginx/access.log or /var/log/apache2/access.log. To rank the busiest AI bots in the last log, run:

• Count hits by bot: grep -Ei "GPTBot|ClaudeBot|CCBot|PerplexityBot|Bytespider|Google-Extended|Amazonbot" access.log | awk '{print $12}' | sort | uniq -c | sort -rn
• See total requests per user-agent: awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
• Estimate bandwidth a bot pulled: grep -i "ClaudeBot" access.log | awk '{sum+=$10} END {print sum/1024/1024 " MB"}'

Watch for these names specifically: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Google's AI training opt-out token), CCBot (Common Crawl, feeds many models), PerplexityBot, Amazonbot, and Bytespider (ByteDance/TikTok — notorious for ignoring robots.txt). If one of these is pulling hundreds of megabytes a day, you've found your problem.

One warning: user-agents can be faked. A polite bot uses its real name and comes from a published IP range. If you see "Googlebot" from a random hosting IP, it's a scraper wearing a costume — those get blocked by IP, not by name.

Slow them down without blocking Google

Start with robots.txt to ask nicely, then enforce with server rules for the bots that don't listen — and never put Googlebot or Bingbot in either list.

Well-behaved AI crawlers honour robots.txt. This is your first, cheapest line of defence. Add a block like this to the file at your web root:

• User-agent: GPTBot then Disallow: /
• User-agent: Google-Extended then Disallow: /
• User-agent: CCBot then Disallow: /
• User-agent: anthropic-ai then Disallow: /

Note that Google-Extended only controls AI training use — it does not affect Googlebot or your search ranking. That's the point: you can opt out of being training data while staying fully indexed. Perplexity and OAI-SearchBot are the ones worth keeping if you value the referral clicks; decide per site.

For the bots that ignore robots.txt (looking at you, Bytespider), enforce at the server. On Apache or a cPanel account, drop this into .htaccess:

• RewriteEngine On
• RewriteCond %{HTTP_USER_AGENT} (Bytespider|CCBot|ClaudeBot) [NC]
• RewriteRule ^ - [F,L]

On an Nginx VPS, add this inside your server block and reload:

• if ($http_user_agent ~* (Bytespider|CCBot|ClaudeBot)) { return 403; }

Do not add search engines to these rules. A 403 to Googlebot is a fast way to fall out of the index.

Throttle instead of block when you're not sure

When a bot brings some value but too much load, rate-limit it rather than banning it — you keep the upside and cap the cost.

On a VPS running Nginx, use limit_req_zone to cap requests per second from a matched user-agent. Map the bots into a variable, then apply a limit only to them, leaving human traffic untouched. A crawler that gets a steady stream of 503 responses simply slows its own crawl rate — that's the intended behaviour, not an error.

On shared hosting you don't get Nginx config, but you have other levers. Crawl-delay in robots.txt (e.g. Crawl-delay: 10) is honoured by Bing and some others, though Google ignores it — set Google's crawl budget in Search Console instead. Caching is your quiet hero here: a well-warmed page cache means a bot hitting the same URL 40 times costs you almost nothing, because it's served from cache instead of firing up PHP and your database each time.

Here's a quick decision guide:

The Cloudflare layer for the ones that lie

Put a free Cloudflare (or similar) proxy in front of your site and you can block bots by verified identity and IP, not just by the name they claim.

The problem with .htaccess rules is that they trust the user-agent string. A proxy layer solves that. Cloudflare's free plan includes a one-click "Block AI Bots" toggle under Bot Management, plus its AI Labyrinth feature that feeds misbehaving scrapers junk pages instead of your real content — which discourages them without costing you real bandwidth. You can also write firewall rules that match verified bot categories.

This is the layer that catches the fakers. A scraper pretending to be Googlebot from a datacentre IP gets caught because the proxy checks the request against Google's published address ranges. For a small site that's being genuinely hammered, moving DNS to a proxy is often less fiddly than maintaining a growing blocklist by hand.

When bot traffic means it's time to move up a tier

If you're regularly blocking and throttling and your shared plan still chokes, the honest answer is you've outgrown shared hosting — and a small VPS gives you the control this problem needs.

Shared hosting is fine until a crawler storm pushes you past your CPU or process limits and your neighbours' sites (and yours) slow down. On a VPS you get root, so you can run limit_req, install fail2ban to auto-ban IPs that hammer you, and tune your PHP-FPM pool to survive a burst. That's a real step up in what you can defend against.

At TPC Hosting we host inside the EU and keep things GDPR-friendly, and our support is staffed by real engineers around the clock — so if a bot surge is knocking your site over at 2am, you're talking to someone who can actually read your logs with you. Migration to a VPS is free, and you've got 30 days to back out if it's not the right fit. Pricing is plain and honest; no surprises when the invoice lands.

Before you upgrade, though, try the steps above. A tight robots.txt, a couple of server rules and a decent cache solve this for most sites without spending a cent.

FAQ