robots.txt Mistakes That Hurt Indexing (and How to Avoid Them)
Disallow is not noindex, blocked CSS can break rendering, and a starter file needs editing before you upload it. Seven common robots.txt mistakes, with a worked example.
A robots.txt file is a few lines of plain text, which is exactly why mistakes in it go unnoticed. One wrong line can stop crawlers reaching your whole site, and a well-meant line can fail to do what you expected. This guide covers the mistakes that cause the most trouble, shows what the robots.txt generator on this site actually outputs, and lists what you still have to fix by hand.
What robots.txt does, in one paragraph
robots.txt sits at the root of your domain (for example https://shop.example.in/robots.txt) and asks well-behaved crawlers which paths they should not fetch. It controls crawling, not indexing. It is also public, so it is not a way to protect anything. Keep those two points in mind; most of the mistakes below come from forgetting one of them.
Mistake 1: Using Disallow to keep a page out of Google
This is the most common one. If a page is blocked in robots.txt, Google cannot read it, so it also cannot see a noindex instruction on that page. If other sites link to the blocked URL, the bare URL can still appear in results, usually without a useful description.
To keep a page out of search results, leave it crawlable and add a noindex robots meta tag (the meta tag generator can write the tag; change its default index, follow to noindex). Only after the page has dropped out should you consider blocking it, and often you will not need to. Google does not honour a noindex line written inside robots.txt itself, so do not rely on one.
Mistake 2: Blocking CSS, JavaScript or image folders
Search engines render pages much as a browser does. If you block /assets/, /static/ or /wp-content/themes/, the crawler may see an unstyled or empty page and judge it poorly, especially on mobile layouts and script-driven sites. Block only what you have a reason to block, such as internal search results or cart pages, and leave your styling and script files open.
Mistake 3: Blocking everything by accident
These two lines look similar and do opposite things:
Disallow:with nothing after it means nothing is blocked.Disallow: /means the entire site is blocked.
The classic version of this mistake is a staging site with Disallow: / that gets copied to the live site at launch. After every deployment, open /robots.txt in a browser and read it. It takes ten seconds.
Mistake 4: Uploading a starter file without editing it
Our generator is a starting point and is deliberately simple. If you type shop.example.in into it, the output is exactly this:
| Line | What it does |
|---|---|
User-agent: * | Applies to all crawlers |
Allow: / | Allows everything by default |
Sitemap: https://shop.example.in/sitemap.xml | Points to your sitemap, assumed to be at /sitemap.xml |
Disallow: /admin/ | A fixed example rule |
Three things need your attention before you upload it:
- The
/admin/rule is a placeholder. If your login area is/wp-admin/,/dashboard/or something else, the rule protects nothing. Replace it with real paths or delete it. - The sitemap URL is a guess. If your sitemap lives elsewhere, or you use a sitemap index, edit the line.
- The Disallow line comes after a blank line. Google treats it as part of the same group, but some simpler parsers read a blank line as the end of a group. The safer layout puts the Disallow rules directly under
User-agentand the Sitemap line at the end.
Mistake 5: Typing the domain with https:// into the generator
The tool adds https:// itself. If you enter https://shop.example.in, the sitemap line becomes Sitemap: https://https://shop.example.in/sitemap.xml, which is not a valid address. Enter the bare domain only, and check the output line before copying it.
Mistake 6: Treating robots.txt as security
Anyone can open your robots.txt. Listing Disallow: /private-reports/ tells every visitor that the folder exists. Well-behaved crawlers follow the file; nothing forces other bots or people to. Protect private content with a login or server-side access rules, and keep it out of robots.txt entirely.
Mistake 7: Expecting rules the file cannot express
Three details trip people up:
- Path matching is case-sensitive.
Disallow: /Private/does not block/private/. - When Allow and Disallow both match a URL, the more specific (longer) rule wins in Google's implementation. That lets you block a folder but allow one file inside it, for example
Disallow: /downloads/followed byAllow: /downloads/brochure.pdf. - Crawl-delay is ignored by Google. Some other search engines read it, but it will not slow Googlebot down.
Also note the file size limit: Google processes the first 500 KiB of a robots.txt file and ignores the rest. A small business site will never get near that, but a generated file with thousands of lines can.
Worked example: fixing a small shop's file
Suppose a small online shop at shop.example.in has product pages at /products/, a cart at /cart/, internal search at /search?q=... and an admin area at /manage/. They paste the generator's output unchanged. Problems:
- The only rule blocks
/admin/, which does not exist, so cart and search pages are still crawlable and waste crawl time. - The real admin area at
/manage/is not mentioned (which is fine for crawling, but it should be behind a login regardless).
An edited version would read:
User-agent: *Disallow: /cart/Disallow: /searchAllow: /Sitemap: https://shop.example.in/sitemap.xml
The cart and internal search results are kept out of crawling, product and category pages stay open, CSS and scripts are untouched, and the sitemap line is at the end. Then generate the sitemap with the sitemap generator, listing only pages you want indexed, and submit it in Google Search Console. After any change, test a few important URLs to confirm they are not blocked.
A short checklist before you upload
- Is
Disallow: /absent from the live file? - Do the blocked paths exist on your site, and are they ones you really want uncrawled?
- Are CSS, JavaScript and images open?
- Does the Sitemap line show a valid https address that returns your sitemap?
- Are pages you want removed from search handled with
noindexinstead of Disallow? - Is the file served at the root, as
/robots.txt, on every hostname you use?
Limits: what this tool and this guide do not cover
- The generator outputs one fixed template. It has no fields for specific crawlers, several Disallow paths, wildcards or crawl-delay; you edit the text yourself.
- It does not test your live file or check whether a URL is blocked. Use the testing and reporting features in Google Search Console for that.
- It does not write noindex tags or decide which pages should be indexed. That is a judgement about your content.
- Rules differ slightly between search engines. This guide describes Google's behaviour; check other engines' documentation if they matter to you.
- A correct robots.txt does not make a page rank. It only avoids stopping crawlers from doing their job.
Tools used in this article
- robots.txt Generator — Enter your domain to generate a starter robots.txt with a sitemap reference — a fixed, standard rule set you can customize further by hand.
- Sitemap Generator — Paste a list of your page URLs and get back a valid XML sitemap, ready to upload and submit to Search Console.
- Meta Tag Generator — Enter your page title, description, keywords, and URL to generate a bundle of core meta tags (description, keywords, Open Graph title/description, robots) in one go.