Four ways robots.txt quietly ruins things
robots.txt controls crawling, not indexing. That single distinction explains most of the damage it causes.
1. Blocking a page to remove it from search
The instinct is reasonable and the result is the opposite of the intention. Disallow stops Google fetching the page — which means Google cannot see the noindex tag on it either.
The page stays in the index, now as a bare URL with no title and no description, sometimes for a very long time.
To remove a page: allow crawling, add noindex. To remove it fast: use the removal tool, then noindex.
2. Disallow: / reaching production
The staging file, deployed. It blocks everything, and unlike a noindex it is visible in one HTTP request — which makes it the easiest of all these to test for and one of the most common to ship.
Worth knowing: a Disallow: / under a named user-agent blocks only that crawler. A file blocking CCBot and ClaudeBot while allowing * is not a blocked site, and reading the file without paying attention to groups produces a false alarm.
3. Blocking assets
Blocking /assets/, /js/ or /css/ prevents Google rendering the page as a user sees it. It renders a version without styling or scripts and assesses that.
This was common practice years ago and is now actively harmful. Google needs those files.
4. Assuming it is private
robots.txt is a public file listing paths you would prefer not to be crawled. It is the first place anyone looks when curious about what a site is hiding.
Disallowing /admin-panel-v2/ publishes its existence to everyone. Use authentication for things that need protecting.
What it is genuinely good for
Declaring your sitemap. Preventing crawl waste on faceted URLs and infinite calendars at scale. Keeping crawlers out of genuinely worthless sections that would otherwise consume crawl budget.
That is roughly the whole list. It is a crawl-management tool, and every attempt to use it as an index-management tool goes wrong.