robots.txt mistakes that quietly hide your pages from Google

robots.txt tells crawlers which parts of your site they may fetch. Because it is short and rarely checked, a single wrong line can quietly block Google from your pages for weeks before anyone notices. The usual culprits are a leftover Disallow: /, blocked CSS and JavaScript, and using robots.txt to “remove” pages when you needed noindex instead.

How robots.txt works, briefly

The file lives at the root of your host, for example https://www.example.com/robots.txt. It contains groups of rules, each starting with a User-agent line:

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://www.example.com/sitemap.xml

A few rules that catch people out:

• Each host and protocol has its own file. shop.example.com doesn’t use the rules from www.example.com.
• Paths are case-sensitive. Disallow: /Blog/ doesn’t block /blog/.
• A crawler follows the most specific group that names it. If there is a User-agent: Googlebot group, Googlebot ignores the User-agent: * group entirely.
• For Google, when an Allow and a Disallow both match, the longer (more specific) rule wins.

Mistake 1: the staging file goes live

Staging and development sites often carry this, to keep them out of search:

User-agent: *
Disallow: /

That is fine on staging. The trouble starts when the file is copied to the live site during a launch or migration. Every page is now off-limits to every crawler. Rankings don’t vanish overnight, so the drop can look like an algorithm change rather than a typo.

Fix: keep robots.txt out of the files you deploy, or generate it per environment. Better still, protect staging with a password, which keeps out people as well as bots. After every launch, open /robots.txt on the live site and read it.

Mistake 2: blocking CSS, JavaScript and images

Older advice said to block folders like /wp-includes/, /assets/ or /js/ to “save crawl budget”. Google renders pages much like a browser. If it can’t load your CSS and JavaScript, it may see a broken or empty page, and it may not see content that scripts add.

Fix: allow everything a page needs to render. Use the URL Inspection tool in Search Console, run a live test, and look at the screenshot and the list of resources that couldn’t be loaded.

Mistake 3: using robots.txt to remove pages from Google

This is the most misunderstood one. Disallow stops crawling. It doesn’t stop indexing. If other pages link to a blocked URL, Google can still list it, usually with no description. Search Console calls this “Indexed, though blocked by robots.txt”.

Worse, if you add a noindex tag to a page and also block it in robots.txt, Google can’t fetch the page, so it never sees the noindex.

Fix: to keep a page out of search results:

1. Add <meta name="robots" content="noindex"> (or an X-Robots-Tag: noindex header).
2. Leave the page crawlable until it drops out of the index.
3. Only then, if you want, block it in robots.txt.

For content that must be private, use a password or remove it. robots.txt is public, so listing secret folders in it just advertises them.

Mistake 4: wildcard and prefix surprises

Google and Bing support two special characters: * matches any sequence of characters, and $ marks the end of the URL. Rules also match by prefix, which surprises people:

Fixes:

• Add a trailing slash to block a folder only: Disallow: /blog/.
• Use $ to match an exact ending: Disallow: /*.pdf$.
• Before blocking all parameters, check which parameter URLs you actually want in search.

Not every crawler supports * and $, so keep important rules simple.

Mistake 5: errors and forgotten groups

If /robots.txt returns 404, Google treats the site as having no restrictions. If it returns a server error (5xx), Google may treat the whole site as blocked until the file comes back. A firewall or CDN rule that challenges bots on /robots.txt can have the same effect. Make sure the file always returns 200 quickly, and is not behind a bot challenge.

Named groups cause the other quiet problem. Adding a group like User-agent: GPTBot is fine, but remember the “most specific group” rule. If you give Googlebot its own group to block one folder, you must repeat any general rules in that group too. For AI-specific rules, see robots.txt rules for AI bots.

How to test before and after a change

• Search Console → Settings → robots.txt shows the version Google last fetched and any parsing problems.
• URL Inspection tells you whether a specific URL is blocked by robots.txt.
• Bing Webmaster Tools has its own robots.txt tester.
• Google caches the file, usually for up to a day, so changes take a little while to apply.
• Keep the old version, so you can roll back quickly.

If pages are missing from Google and robots.txt looks fine, work through why a page isn’t on Google.

Checklist

• Live robots.txt has no Disallow: / you didn’t mean.
• CSS, JavaScript and images are crawlable.
• Pages you want out of search use noindex and stay crawlable.
• Folder rules end with /; exact endings use $.
• The file returns 200 and isn’t behind a bot challenge.
• Named bot groups repeat the general rules they still need.
• A Sitemap: line points to your sitemap.