SEO ToolsTechnical SEOFree Tool
robots.txt Generator
Enter your domain to generate a starter robots.txt with a sitemap reference — a fixed, standard rule set you can customize further by hand.
How to Use robots.txt Generator
- 1Enter your domain name
- 2Click Generate
- 3Review the Allow/Disallow rules and edit if you need to block additional paths
- 4Save the result as robots.txt at your site root
About robots.txt Generator
What the output is
Type your domain (for example example.com) and the tool outputs a starter robots.txt that allows all crawlers, includes a Sitemap: line pointing to https://your-domain/sitemap.xml, and contains a Disallow: /admin/ rule.
Check before you upload it
- The
/admin/rule is a fixed example. Replace it with paths that actually exist on your site, or delete it. - The sitemap location is assumed to be
/sitemap.xml. Change it if yours lives elsewhere. - The tool has no fields for particular crawlers, crawl delay or multiple Disallow paths, so edit the text by hand for those.
What robots.txt can and cannot do
- It asks well-behaved crawlers not to fetch certain paths. It is not a security measure; blocked URLs are public and listing a private path advertises it.
- Blocking a page in robots.txt does not reliably keep it out of search. A blocked URL can still be indexed if other sites link to it. Use a
noindextag on a crawlable page for that. - Never block your CSS or JavaScript files, since search engines need them to render pages.
The file must be served at /robots.txt on the root of the domain. Generate the matching sitemap next.
Frequently Asked Questions
A fixed template: "User-agent: *", "Allow: /", your Sitemap: line, and "Disallow: /admin/". Only the domain (used in the Sitemap: line) is generated from your input.
Not directly from this generator — the Allow/Disallow rules are fixed. Add extra Disallow: lines to the output manually for any other paths you want to block.
Save it as robots.txt in your site's root directory (e.g. https://yoursite.com/robots.txt) so crawlers can find it automatically.
No — the default template allows crawling of everything except /admin/. Double-check it against the Robots.txt Analyzer before publishing if you're unsure.