Robots.txt Tester

What this tester checks

Enter a website and we fetch its robots.txt from the top of the site, following up to five redirects. Or switch to paste mode and drop in the text of a file you have not published yet. Then give a path like /blog/ or a full address, and pick one crawler or all of the common ones.

For one crawler you get a plain Allowed or Blocked, the group of rules that applies to it and the exact rule that decided, with its line number. For all crawlers you get a table with one row per crawler, covering Google, Bing, Apple, DuckDuckGo, Yandex, Baidu and the main AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.

Fetching runs on our servers, so your browser only talks to tpc-hosting.com. We read at most 512 KiB of robots.txt, and we keep nothing. Pasted text stays in your browser and is never added to the page address.

How crawlers read robots.txt

A robots.txt file is split into groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the group that names it. If none does, it uses the group for *. If two groups name the same crawler, their rules are merged. Google also lets a specific crawler such as Googlebot-Image fall back to the Googlebot group.

Inside the chosen group, the rule with the longest matching path wins, and Allow wins when two rules match equally well. A * in a path matches any run of characters, and a $ at the end means the address must end there. An empty Disallow line blocks nothing. The /robots.txt file itself is always allowed.

Paths are compared after normalising percent-encoding, so %7E and ~ count as the same character, while rules are case-sensitive. This tester follows RFC 9309, the standard that Google and other major crawlers implement.

What the HTTP status of robots.txt means

If robots.txt answers with a success status, its rules apply. If it answers with a client error such as 404, crawlers behave as if there is no file and treat everything as allowed. If it answers with a server error such as 503, or cannot be reached at all, crawlers are told to assume the whole site is off limits until the file works again.

That last case catches people out. A broken server or a firewall that blocks crawlers from robots.txt can quietly stop a whole site from being crawled, even when the file itself looks fine in your browser. The tester shows the status first, so you can see this before you read the rules.

Common robots.txt mistakes

Blocking a page in robots.txt does not remove it from search results. It stops crawlers from reading the page, so a blocked address can still be listed without a description if other sites link to it. To keep a page out of search, leave it crawlable and use a noindex tag instead.

Other frequent slips are a rule written without a leading slash, a Disallow placed before any User-agent line, a typo in a directive name, and a stray Disallow: / left over from a staging site. The tester lists these as warnings with their line numbers. To write a clean file from scratch, use our robots.txt generator.