Robots.txt Generator
This free robots.txt generator builds a valid robots.txt file for your site, with your own allow and block rules, sitemap links and a switch for each major AI crawler.
Your robots.txt
Built in your browser. Nothing you enter is sent to us. Upload the file to your site's root folder so it loads at /robots.txt.
What is robots.txt?
A robots.txt file is a plain text file at the root of your site (example.com/robots.txt) that tells crawlers which parts of the site they may visit. RFC 9309 made the format an official standard in 2022.
Each group starts with one or more User-agent lines naming crawlers, followed by Disallow and Allow rules. A crawler follows the most specific group that names it, and within a group the longest matching rule wins.
robots.txt is a request, not a lock
Well-behaved crawlers like Googlebot follow robots.txt, but nothing forces anyone to. Never rely on it to hide private pages. Protect those with a password instead; our htpasswd generator can help.
Blocking a page also doesn't take it out of Google if other sites link to it. To keep a page out of search results, let Google crawl it and add a noindex tag.
Should you block AI crawlers?
It depends on what you want from AI tools. Training crawlers like GPTBot, ClaudeBot and CCBot collect pages for future models. Search and assistant crawlers like OAI-SearchBot, PerplexityBot and Claude-User fetch pages so AI answers can quote and link them.
Blocking training crawlers keeps your content out of future models, but not out of ones already trained. Blocking search crawlers makes AI answers less likely to cite or link you, which can cost you visitors. A common middle ground is to block training and allow search.
Crawl-delay and sitemaps
Crawl-delay asks a crawler to wait a number of seconds between requests. Google ignores it and sets its own pace based on how quickly your server responds, while some other crawlers, such as Bing, respect it.
Sitemap lines point crawlers at your XML sitemap. They must be full URLs, and you can list more than one. Everything here is built in your browser, so nothing you type is sent to us.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your website, so it loads at https://yourdomain/robots.txt. On most hosting plans that's the public_html folder. Each subdomain needs its own file.
What's the difference between Disallow: and Disallow: /?
An empty Disallow: allows everything. Disallow: / blocks the whole site. That one slash is the most common robots.txt mistake, so check it before you upload.
Will blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. AI Overviews are part of Google Search and follow Googlebot's rules.
Do all AI crawlers obey robots.txt?
Not necessarily. The large AI companies say their named crawlers follow it, but robots.txt can't force anyone, and some crawlers have been reported to ignore it. To stop a crawler for certain, block it on your server or firewall as well.
Does robots.txt affect SEO?
Indirectly. It controls what search engines can crawl, so a wrong Disallow can hide your whole site from Google. It doesn't improve rankings on its own.
How quickly do changes take effect?
Google generally caches robots.txt for up to 24 hours, so give it a day. Other crawlers fetch it on their own schedules.