Back to Article List

AI Agents Skip Your Checkout: Read Your Own Bot Logs

Agenții AI ocolesc checkout-ul: analizează logurile de bot - AI Agents Skip Your Checkout: Read Your Own Bot Logs

Recent bot-traffic data showed agents from two big AI companies accounting for roughly 90% of automated requests — and they were pulling product and search pages while barely touching checkout. If your store has felt busier in the logs but not in sales, that's probably why: a lot of your "traffic" is machines reading, not buying.

The good news is you don't have to take anyone's word for it. Your own access log already has the answer. Twenty minutes of grepping will tell you which agents are visiting, what they hit, what they skip, and whether they're wasting your server's time on pages you could serve from cache instead.

Here's how to pull that picture, read it honestly, and act on it without accidentally blocking the bots you actually want.

Where your bot traffic actually shows up

Every request to your site lands in an access log — on most Linux hosts that's /var/log/nginx/access.log or /var/log/apache2/access.log, and in cPanel it's under Metrics > Raw Access. That one file tells you the visiting IP, the URL requested, the response code and the user agent string, which is where bots announce themselves.

AI agents don't hide the way scrapers used to. OpenAI's crawlers identify as GPTBot, OAI-SearchBot and ChatGPT-User; Meta's show up as meta-externalagent and facebookexternalhit. Anthropic sends ClaudeBot. Perplexity uses PerplexityBot. The agentic ones — the kind that fetch a page on a user's behalf rather than crawling for an index — tend to carry the "-User" or "agent" markers and arrive in short, bursty sessions rather than steady crawls.

If you're on TPC Hosting, your raw logs sit in your hosting panel and over SSH, so you can run the commands below directly instead of relying on a third-party analytics dashboard that already filtered the bots out before you saw them.

Pull the picture in a few commands

To see who's really hitting your site, count requests by user agent and then break down what those agents asked for. Start with the top talkers:

This ranks user agents by request count:

  • awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

Then isolate a single agent and see which paths it hit. Swap in whichever name dominated the list above:

  • grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

Now the part that matters for a store. Count how often the agent touched product and search pages versus checkout and cart:

  • grep "meta-externalagent" access.log | grep -cE "/product|/shop|/category"
  • grep "meta-externalagent" access.log | grep -cE "/checkout|/cart|/basket"

If the first number is in the thousands and the second is near zero, you've just confirmed the pattern on your own server. The agent is reading your catalogue, not converting. That's not a bug to panic over — it's a signal that tells you exactly how to tune your caching.

What the "skip checkout" pattern actually means

Agents skip checkout because they can't log in, can't fill a payment form and usually stop at the point where a real decision or authentication is needed. So the heavy traffic is read-only, repetitive, and almost entirely on pages that don't change per visitor — which is the best possible news for your server load.

This splits your site neatly into two zones. Product listings, category pages, individual product pages and your blog are public and cacheable: the same HTML for everyone, perfect for serving from cache so the bot never touches PHP or your database. Cart, checkout, account and search-with-parameters are dynamic and personal: you don't want those cached, and you don't want bots hammering them either.

The mistake is treating all bot traffic as an attack. A crawler reading your product pages from a full-page cache costs you almost nothing and may put your products in front of people asking an AI assistant for recommendations. The same crawler generating thousands of uncached search queries — each one a fresh database hit — is a different story.

What to cache, what to throttle

Cache everything public and static aggressively; throttle anything that forces dynamic work per request. Here's the split for a typical small store:

Page typeAgent behaviourAction
Product & category pagesHeavy, repeated readsFull-page cache, long TTL
Blog / contentCrawled for indexingFull-page cache
Internal search (?s= / ?q=)Bursty, uncacheable, DB-heavyThrottle or block for bots
Faceted filters (?color=&size=)Infinite URL combinationsBlock via robots.txt + rate limit
Cart / checkout / accountRarely touched by agentsNever cache; rate limit

The quiet killer on most stores is faceted navigation. Every filter combination is a unique URL, and a crawler will happily walk all of them — thousands of near-identical pages, each one a database query. Block those parameter URLs in robots.txt with lines like Disallow: /*?color= and Disallow: /*?s=, and the problem mostly evaporates for the well-behaved bots.

For WordPress and WooCommerce, make sure your caching plugin is set to never cache cart, checkout, my-account and any URL with a cart cookie — most do this by default, but confirm it. On a managed TPC plan the server-side cache handles the public pages before the request ever reaches PHP, which is exactly where you want it for this kind of read-heavy bot load.

Throttling without blocking the bots you want

Rate-limit by request rate per IP rather than banning agents outright, so a polite crawler sails through while an aggressive one gets slowed down. A hard block on GPTBot or meta-externalagent also removes you from the AI answers those tools generate — which, for a small store, may be free visibility you don't want to throw away.

On Nginx, a simple limit in the http block keeps any single IP from flooding you:

  • limit_req_zone $binary_remote_addr zone=bots:10m rate=10r/s;
  • then inside your location block: limit_req zone=bots burst=20 nodelay;

If you'd rather manage the bad actors specifically, Cloudflare's free tier lets you set a rate-limiting rule or challenge requests that claim to be a crawler but come from the wrong network. Verify the real ones first: legitimate Googlebot, GPTBot and others publish IP ranges or support reverse-DNS checks, and plenty of "GPTBot" strings in the wild are fakes riding the name.

Quick checklist before you change anything:

  • Confirm the pattern in your own logs — don't act on headlines alone.
  • Cache public pages hard; exclude cart, checkout and account by cookie.
  • Block parameter URLs (search, filters) in robots.txt.
  • Rate-limit per IP instead of banning user agents.
  • Verify any bot you're tempted to block is who it claims to be.
  • Re-check the logs a week later to see what changed.

If digging through log files and cache rules isn't how you want to spend your week, that's a fair place to ask for help. TPC Hosting's chat has AI answers at any hour with real engineers following up in business hours, so you can get a second pair of eyes on your access log and your cache settings before you throttle something you shouldn't.

The takeaway for small stores

Heavy AI-agent traffic that skips checkout isn't a threat to defend against — it's a load pattern to tune for. Serve your public pages from cache, keep the dynamic ones protected, block the parameter sprawl, and rate-limit the greedy rather than banning the useful.

Do that and the same traffic that looked alarming in your analytics becomes nearly free to serve. Your server stays responsive for the humans who do reach checkout, and your products stay visible to the assistants more people are using to shop.

FAQ

How do I tell an AI agent apart from a regular search crawler in my logs?

Check the user agent string: agentic fetchers usually carry markers like ChatGPT-User or meta-externalagent, while indexing crawlers use names like GPTBot or Googlebot. Agents also arrive in short bursts tied to a user query, whereas crawlers visit steadily over time.

Should I block GPTBot and Meta's agents entirely?

Usually not — blocking them also removes your products from the AI answers those tools generate, which can be free visibility for a small store. Rate-limit aggressive behaviour per IP instead, and only block outright if a bot is ignoring robots.txt or faking its identity.

Why are AI agents hitting my product pages but never my checkout?

Agents can't log in or complete a payment form, so they stop at public, read-only pages like products, categories and search. That means the bulk of their traffic is cacheable, which is good news for your server load once you tune caching correctly.

What's the single biggest caching mistake that lets bots overload a store?

Leaving faceted-filter and internal-search URLs uncached and crawlable, because each parameter combination is a unique page that triggers a fresh database query. Block those URL patterns in robots.txt and the load from well-behaved bots drops sharply.

Where do I find my raw access logs?

On most Linux servers they live in /var/log/nginx/ or /var/log/apache2/, and in cPanel they're under Metrics then Raw Access. With TPC Hosting you can reach them through the hosting panel or over SSH to run log analysis commands directly.