# Robots.txt Generator

Build a valid robots.txt without looking up the syntax: pick how open the site should be, add the paths crawlers should leave alone, name your sitemap, and copy or download the result. It is generated in your browser, and one click publishes it to a live link so you can see exactly what a crawler will read.

This tool runs entirely in the browser. Nothing is uploaded, and no account is needed.

## How to use it

1. **Pick how open the site is.** Start from allow everything, block everything, or allow with exceptions, and the base rules are written for you.
2. **Add your rules.** List the paths crawlers should leave alone, add a group for a specific bot if you need one, and name your sitemap.
3. **Copy, download or publish.** Copy the file, download it as robots.txt, or publish it to a live link to see exactly what a crawler will read.

## Frequently asked questions

### Is the robots.txt generator free?

Yes, free with no sign-up, and the file is generated in your browser rather than on a server.

### Where does the file have to go?

At the root of the host, as /robots.txt. It does not work from a subfolder, and every subdomain needs its own copy.

### Does Disallow hide a page from search results?

No. It stops a crawler fetching the page, but a disallowed URL can still be listed with no description if other pages link to it. Use a noindex meta tag on a page that is allowed to be crawled.

### Can I block one crawler and allow the rest?

Yes, by adding a group that names it. Remember that a crawler obeys only the most specific group that matches it, so any rule it still needs has to be repeated in that group.

### Do I need a Sitemap line?

It is optional and worth adding. It must be an absolute URL, and it applies to the whole file rather than to one user-agent group.

### How do I test the result?

Click Publish to put the generated file on a live HTTPS link and open it, so you see exactly the bytes a crawler would read before you deploy it to your own host.