robots.txt Generator
Add rules per crawler (user-agent) with Disallow and Allow paths, add your sitemap URLs, and get a robots.txt file for your site root as you type. Presets cover search engines such as Googlebot and Bingbot as well as AI crawlers like GPTBot, ClaudeBot, CCBot, and Google-Extended. For the rest of your search-facing markup, see the Meta Tag & OGP Generator.
🔒 Runs in your browser · Your data is never sent to a server
Presets
Click to add a group (it replaces the initial group if that is still empty).
Rules (groups)
Sitemap
Checks
How to use
- Add a preset that fits your goal, or click "Add group" to write your own rules.
- For each group, enter the user-agents and the Disallow and Allow paths, one per line. Paths must start with /.
- Optionally add Sitemap URLs, one per line.
- Review the warnings, then copy the result or download robots.txt. Place it at the root of your site (/robots.txt).
Notes
- robots.txt is a request to crawlers, not access control. Anyone can read it and some bots ignore it. Use authentication to protect anything confidential.
- A URL blocked with Disallow can still appear in search results if other pages link to it. To keep a page out of search, use noindex (and allow crawling so the noindex can be read).
- Crawl-delay is honored by some crawlers such as Bing, but Googlebot ignores it.
- An empty `Disallow:` blocks nothing, while `Disallow: /` blocks the entire site.
- Serve the file as UTF-8 text at the root of the host (e.g. https://example.com/robots.txt). Each subdomain needs its own file.
Glossary
User-agent
The name of a crawler. `*` matches every crawler, while a name such as Googlebot or GPTBot targets only that crawler.
Disallow / Allow
Path prefixes crawlers must not fetch / may fetch. They start with `/`, and the more specific (longer) rule wins.
Sitemap
A line that tells crawlers where your sitemap.xml lives. Use an absolute URL; multiple lines are allowed.
FAQ
Where do I put robots.txt?
At the root of the host, for example https://example.com/robots.txt. A file in a subdirectory has no effect, and every subdomain needs its own file. On a static site, save it as robots.txt at the top of your public folder.
How do I block AI training crawlers?
Name each crawler (GPTBot, ClaudeBot, CCBot, Google-Extended, and so on) as a User-agent and add Disallow: /. The "Block AI bots" preset adds them all at once. Compliance is voluntary on the crawler operator's side, so this is a request, not enforcement.
Why does a Disallowed page still appear in search results?
robots.txt only limits crawling, not indexing. If other pages link to the URL, it can still be listed without a snippet. To remove a page from search, use noindex and leave the page crawlable so the directive can be read.
Which wins when Allow and Disallow conflict?
Google and Bing apply the most specific (longest) matching path. With Disallow: /admin/ and Allow: /admin/public/, only /admin/public/ and below can be crawled.
Share this page