Bot Traffic Is Now a Hosting Bill: Cut the Noise Yourself

A big slice of the traffic hitting your site right now is not human, and you are paying for it in bandwidth, CPU, and slower response times for the people who actually buy from you. The fix is not a magic button. It is thirty minutes with your access logs to see who is really knocking, then a few targeted rules that block the freeloaders without touching Google or your customers.

Two things pushed this into the open recently: Kinsta published a report crunching over 10 billion requests showing how much of the web is now bots, and Cloudflare shipped new tooling aimed squarely at AI crawlers. The takeaway for the rest of us is simpler than the headlines. You do not need to buy a bot-defence platform to get most of the benefit. You need to read your own signals and act on them.

Here is how to do that, and where a host-level block is enough versus where a CDN rule genuinely earns its place.

Why bot traffic shows up on your hosting bill

Bots cost you money because every request they make burns the same resources a real visitor would, except they never convert, subscribe, or buy anything. A scraper hammering your product pages 40 times a second still triggers PHP, hits your database, and eats bandwidth. On a shared or entry-level plan that is the difference between a snappy site and one that crawls at 2pm.

The traffic breaks into three rough buckets. Good bots you want (Googlebot, Bingbot, uptime monitors you set up). Grey bots you might tolerate (AI training crawlers like GPTBot and ClaudeBot, SEO tools like AhrefsBot). And bad bots you never want (credential stuffers, content scrapers, vulnerability scanners poking at /wp-login.php and /.env).

The mistake most people make is treating all automated traffic as one problem. Block too broadly and you knock yourself out of search results or break a payment webhook. The goal is surgical: cut the noise, keep the signal.

Read your access logs before you touch a firewall

Start by finding out who your top visitors actually are, and you can do this today with tools already on your server. Guessing leads to blocking the wrong things.

SSH into your account and point these at your access log (path varies — often /var/log/nginx/access.log, logs/access.log, or check cPanel's Raw Access logs). First, the busiest IPs:

• Top 20 IPs by request count:
  awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
• Top user agents:
  awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
• What one noisy IP is actually requesting:
  grep "203.0.113.45" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

Numbers alone do not decide anything — the shape of the requests does. Here is what the top-IPs command might spit back:

Now read them like a story. The 51,000-hit IP, when you grep its requests, is pounding a single path — POST /wp-login.php over and over. That is a brute-force attack, block it. The 8,900-hit IP in Google's range walking your sitemap and article URLs at a steady pace is exactly the crawler you want; leave it alone. The 7,400-hit IP with a generic browser user agent requesting /product/1, /product/2, /product/3 in tidy sequence is a scraper harvesting your catalogue — a candidate for a rate limit or a block. The 612-hit IP is likely a real person or a small tool and not worth a second thought.

Grep for the obvious troublemakers too: grep -Ei "wp-login|xmlrpc|\.env|phpmyadmin" access.log | wc -l tells you how much of your traffic is just vulnerability probing. If that number is in the tens of thousands, you have found where your CPU is going.

Verify Googlebot instead of trusting the user agent

Never whitelist or block based on the user-agent string alone, because anyone can claim to be Googlebot in one line of code. The right check is a reverse DNS lookup, and it takes seconds.

Take any IP claiming to be Google and run:

• host 66.249.66.1 — a real Googlebot resolves to something ending in googlebot.com or google.com
• Then confirm the forward lookup matches: host crawl-66-249-66-1.googlebot.com should return the same IP

If the reverse DNS shows a random hosting provider or a residential ISP but the user agent says "Googlebot", that is a fake and safe to block. Google, Bing, and most legitimate crawlers publish their IP ranges or support this DNS verification. This single habit prevents the most common self-inflicted wound: blocking a spoofed bot with a broad rule and accidentally catching the real one.

The grey-bot call most people get wrong

Everyone rushes to block GPTBot, ClaudeBot, and the rest. Slow down on that one. Blocking AI training crawlers is a fair choice if you sell your content or hate the idea of it feeding a model. But there is a second category people conflate with training: the crawlers that fetch pages in real time to answer a user's question and cite you back. If you slam the door on those, you also disappear from AI answer engines that are already sending referral clicks — and that traffic is real people who might buy.

My rule of thumb: block the training crawlers if you feel strongly, but keep the answer-and-cite fetchers open unless your logs show them actually abusing you. Check your referrer data before you decide. If you are seeing visits arrive from AI search surfaces, a blanket block is you turning away customers to make a point. Decide with numbers, not vibes.

When a host-level block is enough

For low-volume nuisances and known-bad IPs, blocking at the server or in your site config is enough and costs you nothing. Reach for a CDN only when the traffic is high-volume, distributed, or changing IPs faster than you can list them.

Host-level tools you already have:

• .htaccess or Nginx — block a user agent or IP range directly. To stop an AI crawler cleanly, a robots.txt rule is the polite first step (User-agent: GPTBot then Disallow: /), but note that only well-behaved bots obey it.
• fail2ban — the workhorse for login attacks. Point a jail at your access log, match repeated hits on /wp-login.php, and it bans the IP at the firewall for a set window. This alone kills most brute-force noise.
• A WAF plugin (Wordfence, or your CMS equivalent) — rate-limits and blocks known patterns without you writing regex.

Here is a simple way to decide:

If you are on TPC Hosting and a block looks like it might catch legitimate traffic, our engineers are on support around the clock and will look at the log with you before you flip a rule that hurts. That second pair of eyes is worth more than any dashboard.

When a CDN rule earns its keep

A CDN like Cloudflare earns its place the moment the bad traffic is distributed across hundreds of IPs, because blocking one address at a time is a losing game there. It also absorbs the load before it ever reaches your server, which is the whole point when a scraper is spread across a botnet.

The rules that pull their weight, in order of usefulness:

• Rate limiting — cap requests per IP to a sane number (say, 60 requests per minute to your login or search endpoints, a starting figure you tune to your own traffic). This stops most scrapers without a single manual block.
• Managed challenge on suspicious scores — let the CDN show a lightweight check to traffic it flags, rather than a hard block. Real people pass it; headless scrapers usually do not.
• Geo or ASN rules — if most of your abuse comes from one hosting provider's ASN and you have no legitimate visitors there, block the ASN, not endless IPs.

Here is what "it worked" looks like in the logs. Before the rate limit, your busiest IP shows 51,204 hits, nearly all to one endpoint. After you set a 60-per-minute cap and give it a day, that same IP tops out around a few hundred hits before every extra request gets a 429, and it eventually gives up and drifts down the list. A good log after tuning is boring: your top entries are Googlebot, your uptime monitor, and a scatter of real browsers, with no single address running away with the request count. A bad log still has one or two IPs an order of magnitude above everyone else — which means your rule missed, usually because the scraper rotated IPs and you need an ASN rule instead of a per-IP cap.

Resist the urge to enable every toggle. Aggressive bot-fight modes can throw challenges at real customers on mobile networks and at legitimate API calls. Start with rate limiting, watch your logs for a week, and tighten only where the noise persists.

A short checklist you can run this week

The whole job fits into an afternoon, and doing it once teaches you more about your traffic than any report will.

• Pull your top 20 IPs and user agents from the access log.
• Reverse-DNS anything claiming to be Google or Bing before trusting it.
• Check your referrers before blocking AI crawlers — keep the ones sending you clicks.
• Add a robots.txt block for AI crawlers you do not want indexing your content.
• Set up fail2ban (or your host's equivalent) on your login endpoint.
• Deny the two or three worst offender IPs at the server.
• Only if abuse is distributed: add a CDN rate-limit rule and a managed challenge.
• Re-check the logs a week later and confirm you did not lose any legitimate crawlers.

You do not need to win an arms race against every bot on the internet. You need to protect your resources and your search visibility, and that comes down to knowing who is knocking and answering accordingly.

Want a hand reading your logs or setting sane limits without breaking search? Talk to the real engineers at TPC Hosting — EU-hosted, GDPR-friendly, and 30 days to back out if it is not the right fit.

FAQ