SEO tools
Robots.txt Generator
Build a valid robots.txt without looking up the syntax: pick how open the site should be, add the paths crawlers should leave alone, name your sitemap, and copy or download the result.
What robots.txt controls, and what it does not
A robots.txt file tells well-behaved crawlers which paths they may fetch. That is the whole of it. It is a request, honoured by the search engines that matter and ignored entirely by anything that does not want to honour it, and the file itself is public: anyone can read yours by adding /robots.txt to your domain. Listing a secret directory in it is a way of publishing the location of that directory.
The distinction that catches people out is between crawling and indexing. Disallow stops a crawler fetching a page; it does not stop that page appearing in results. If other sites link to a disallowed URL, a search engine can list it with no description, because it was told not to look. To keep a page out of results, let it be crawled and serve a noindex robots meta tag or header on it. Disallowing it instead guarantees the engine never sees the instruction.
How the rules are actually matched
A crawler picks exactly one User-agent group: the most specific one that names it, or the * group if none does. It does not combine them. So a group for a named bot completely replaces the general rules for that bot, which means every rule it still needs has to be repeated inside it. This is the single most common mistake in a hand-written robots.txt.
Within a group, paths are prefix matches from the root, * stands for any run of characters and $ anchors the end. Disallow: /admin therefore blocks /admin, /admin/ and also /administration, which is usually not what was meant; Disallow: /admin/ is the safer form. An empty Disallow: means allow everything, and Disallow: / means block the whole site, which are one character apart.
The sitemap line, and where the file goes
A Sitemap: line is independent of the user-agent groups and applies to the whole file, so it belongs at the end. Its value must be a full absolute URL including the scheme and host, because a crawler that finds it may not be the one that fetched your robots.txt. It is the cheapest way to make a sitemap discoverable without submitting it anywhere.
The file only works at the root of a host: https://example.com/robots.txt, never in a subfolder, and a subdomain needs its own. It must be served as plain text. Test it before you rely on it, because a file that accidentally disallows everything looks almost identical to one that allows everything, and the difference is measured in weeks of lost traffic.
How to use it
-
Pick how open the site is
Start from allow everything, block everything, or allow with exceptions, and the base rules are written for you.
-
Add your rules
List the paths crawlers should leave alone, add a group for a specific bot if you need one, and name your sitemap.
-
Copy, download or publish
Copy the file, download it as robots.txt, or publish it to a live link to see exactly what a crawler will read.
Frequently asked questions
Is the robots.txt generator free?
Where does the file have to go?
/robots.txt. It does not work from a subfolder, and every subdomain needs its own copy.Does Disallow hide a page from search results?
noindex meta tag on a page that is allowed to be crawled.