Robots.txt vs WebMCP: Reading and Acting Are Now Split

Your robots.txt controls who reads your site. It does nothing about who acts on it. Those used to be the same problem. They aren't anymore.

A new standard called WebMCP lets AI agents trigger real actions inside a website — adding items to a cart, filling a form, starting a checkout — through tools the site itself exposes. It's already shipping in products from OpenAI, Shopify and Cloudflare. So while everyone tunes their crawler rules to match their AI policy, a second door has opened, and most small sites haven't looked at it yet.

Here's the short version: keep managing crawlers with robots.txt, but stop assuming that file protects anything that does things. Below is what's actually different, and what to lock down first if you run a small site.

Reading and acting are now two separate problems

Blocking a bot from reading your content and stopping an agent from taking an action are two different jobs, handled by two different mechanisms. Getting one right tells you nothing about the other.

Reading is the old game. A crawler requests pages, you allow or disallow paths in robots.txt, and well-behaved bots respect it. The stakes are about content: training data, scraping, showing up (or not) in AI answers. The recent wave of "bot preference" tooling — like Cloudflare's controls and the emerging AI-crawler directives — is all still about reading. It aligns your robots.txt with a policy: yes to Googlebot, no to this AI trainer, maybe to that one.

Acting is the new game. WebMCP (Web Model Context Protocol) is a way for a site to publish a set of tools — small, defined actions — that an AI agent can call on behalf of a user. Think "search products", "add to cart", "book appointment". The agent doesn't scrape a page and guess; it calls the function you exposed. That's genuinely useful. It also means the thing on the other side isn't reading — it's doing.

The mistake to avoid: treating a tightened robots.txt as if it now covers agents. It doesn't. One file governs pages fetched by crawlers. The other governs actions invoked by agents. Neither is aware of the other.

What WebMCP actually changes for a small site

WebMCP turns your website into something an agent can operate, not just read — and that shifts your risk from content to transactions. The moment an action can be triggered by software instead of a person, the questions change from "what did they see?" to "what did they do, and on whose authority?"

Practically, most small sites won't hand-build WebMCP tools tomorrow. You'll get exposed to it through platforms. If you run a Shopify store, agent-driven actions arrive through Shopify. If you sit behind Cloudflare, agent traffic and controls arrive there. If your visitors use ChatGPT's agent features, they may start interacting with your forms and checkout through an intermediary you never configured.

So the realistic near-term picture is not "I need to write MCP tools." It's "actions on my site can now be initiated by an agent acting for a user, and I need my normal defences to hold up when the actor is fast, tireless and non-human." Rate limits, form validation, checkout fraud checks and auth all matter more, because the assumption that a human is slowly clicking through is gone.

Find out what's already exposed before you harden anything

Before you lock doors, find out which ones exist. You can't reason about agent access if you don't know what your platform already hands to agents on your behalf.

Start with the manifest. A WebMCP-style setup advertises its tools somewhere an agent can discover them — often a JSON document listing each tool's name, description and the parameters it accepts. On Shopify, agent-callable commerce actions surface through its storefront and agent APIs rather than a file you edit, so check your admin under app and sales-channel permissions to see what's turned on; anything that can create a cart or start a checkout is a tool an agent can drive. On Cloudflare, look in the dashboard for AI/agent and bot controls — that's where agent traffic gets allowed, challenged or logged, and where you'll see whether an agent path is open.

Then read your own logs, because a manifest tells you what's possible and logs tell you what's happening. The signature to watch for isn't subtle once you know it: repeated, identical action calls from a single session in a tight window, often with a non-browser user-agent (no Mozilla/5.0 string, or an SDK/library name) hitting a state-changing endpoint like /cart/add or /checkout. A human doesn't submit the same form forty times in ninety seconds. An agent will, and it'll do it from one session with clockwork timing. If you see that pattern, you're already being operated, not just read.

What to lock down first

Lock down the actions that cost you money or leak data before you touch anything else — everything that changes state, not just what gets read. Read-blocking is a preference; action-abuse is a bill.

Work through these in order:

• Any endpoint that spends, books or commits. Checkout, booking, quote requests, "contact sales". Rate-limit them per IP and per session. On nginx, a concrete starting point is limit_req_zone $binary_remote_addr zone=actions:10m rate=10r/m; then limit_req zone=actions burst=5 nodelay; on the location — ten requests a minute, small burst allowed. If a form can be submitted 200 times a minute today, an agent will find out.
• Auth on anything that reveals account data. If an action returns order history, saved addresses or invoices, it must require a real logged-in session — never a guessable URL or a token in a query string.
• Server-side validation on every form. Agents don't run your client-side JavaScript checks. Validate and sanitise on the server, every time, no exceptions.
• Bot-friendly friction on abuse-prone forms. A quiet honeypot field plus a cap of, say, three submissions per session per hour stops far more automated junk than a CAPTCHA, without punishing real visitors.
• Logging you can actually read. Log which actions fire, how often, and from where, including the user-agent. You can't spot agent misuse if a spike in checkouts looks identical to a good day.

Notice what's not at the top: your robots.txt. Keep it tidy and aligned with your AI stance, but it protects content, not your cart. Fix the money doors first.

Keep your robots.txt honest anyway

Robots.txt still matters for controlling who reads and trains on your content, so keep it accurate — just don't overload it with expectations it can't meet. It's a public statement of preference that cooperative crawlers honour. That's real value, and it's worth setting deliberately.

A clean starting point for a small site that wants search visibility but wants to opt out of AI training crawlers looks like this:

Two honest caveats. First, robots.txt is voluntary — a badly behaved scraper ignores it, so use server-level blocking or a service like Cloudflare for the ones that don't play fair. Second, blocking AI crawlers can reduce how often your brand surfaces in AI answers. That's a business decision, not a security one. Decide it on purpose.

A quick reality check before you panic

Most small sites are not the target of agent abuse yet, so treat this as good hygiene rather than an emergency. The point isn't fear — it's that the correct defences moved, and it's cheap to move with them.

If your action endpoints already have rate limits, server-side validation and real authentication, you're most of the way there. WebMCP doesn't introduce a brand-new attack; it makes existing weak spots easier to hit at speed. A form with no server-side check was always a problem. An agent just makes the problem punctual.

If you host with us at TPC Hosting, our engineers are on support 24/7 and can help you set sensible rate limits and read your logs for the agent signature above before it costs you. Everything runs EU-hosted and GDPR-friendly, which also keeps you on the right side of the "whose data did that agent just touch?" question. Migration is free and you've got 30 days to back out.

Do the boring things well: guard the actions, log them, and keep robots.txt honest about reading. That combination handles the agent era far better than any single clever setting.

FAQ