robots.txt explained: the small file that can break your whole site's SEO

The robots.txt file is one of the smallest files on your website and one of the most dangerous to get wrong. It lives at the root of your domain, it's often just a few lines long, and it controls which parts of your site search engine crawlers are allowed to request. Used well, it's a quiet housekeeping tool. Used carelessly, a single misplaced line can tell Google to stop crawling your entire site — a mistake that has quietly wiped out traffic for businesses that didn't notice for weeks. This guide explains exactly what robots.txt does, the crucial distinction it's famous for confusing, the mistakes that cause real damage, and how to use it without hurting yourself.

What robots.txt actually controls

robots.txt is a file at the root of your domain that tells crawlers which URL paths they may or may not request. When a well-behaved crawler visits your site, it reads this file first and respects the rules: 'you may crawl these paths, please don't crawl those.' That's its entire function — controlling crawler access to paths. It's useful for keeping crawlers out of areas that don't need crawling: admin sections, internal search results, faceted-navigation URLs that generate endless combinations, and other low-value paths that would otherwise waste crawling. Think of it as a polite sign telling crawlers where not to bother going, which the major engines honour.

The distinction that trips everyone up

Here is the single most important and most misunderstood thing about robots.txt: blocking a URL from being crawled is not the same as keeping it out of the index. If you block a page in robots.txt, Google won't crawl it — but it can still index the URL (often showing it without a description) if it's linked from elsewhere, because it knows the URL exists even though it can't read the page. So robots.txt is the wrong tool for hiding a page from search results. To keep a page out of the index, you need a noindex tag on the page — which, crucially, Google can only see if the page is crawlable. Blocking the page in robots.txt prevents Google from ever seeing the noindex.

The trap of blocking a page you want deindexed

This leads to a classic, counterintuitive mistake. Someone wants a page out of Google, so they add both a noindex tag and a robots.txt block. But the robots.txt block stops Google crawling the page, which means it never sees the noindex tag, which means the page can stay indexed. The two instructions conflict, and the block wins by preventing the noindex from ever being read. The correct way to deindex a page is to let it stay crawlable and add a noindex tag, or remove it and return the right status. Only once a page is reliably deindexed does blocking it in robots.txt make sense, if at all.

The most expensive mistake

The costliest robots.txt error is a single over-broad disallow that blocks far more than intended — most catastrophically, a rule that blocks your whole site. This happens most often during launches and migrations: a staging site legitimately blocks all crawling, and when it goes live, someone forgets to remove the block. The result is a brand-new or newly-migrated site that Google is told not to crawl at all, and traffic that never arrives or quietly collapses. Because robots.txt has site-wide reach, a mistake in it is uniquely damaging, and because nothing on the pages themselves looks wrong, it can go unnoticed for weeks. Always check robots.txt immediately after any launch or migration.

What belongs in robots.txt

Good uses of robots.txt are modest: keeping crawlers out of genuinely low-value or infinite URL spaces (internal search, certain parameter and filter combinations, cart and checkout paths), preventing crawl waste on large sites, and pointing crawlers to your sitemap. The guiding principle is that robots.txt is for managing crawl efficiency on paths that don't need crawling, not for controlling what appears in search results. If your goal is 'don't waste crawling on this,' robots.txt is right. If your goal is 'don't let this appear in Google,' robots.txt is the wrong tool and you want noindex instead, as our guide to getting indexed covers.

What does not belong in robots.txt

Don't use robots.txt to try to hide sensitive content — it's a public file anyone can read, so disallowing a secret path effectively advertises its location. Don't use it to deindex pages (use noindex). Don't block resources like CSS and JavaScript that the engine needs to render your pages properly, which can break how Google sees your content. And don't block your own important sections by accident with an over-broad rule. The file should be minimal and deliberate: every line should have a clear, intended purpose, because in robots.txt the cost of an unintended consequence is unusually high.

How it interacts with sitemaps

robots.txt and your XML sitemap work together and should agree: reference your sitemap's location in robots.txt so crawlers can find it, and make sure you're not blocking URLs that your sitemap lists as important, which is a contradictory signal, as our XML sitemaps guide explains. A sitemap says 'please find and crawl these pages'; a robots block says 'don't crawl these.' If the same URLs appear in both with opposite instructions, you're confusing the engine and undermining both signals. Keeping the two consistent — sitemap lists what you want crawled, robots blocks only what you don't — is part of a coherent technical setup.

Testing before you trust

Because the blast radius is so large, test robots.txt changes before relying on them. Search console includes a robots.txt tester that shows whether a given URL is allowed or blocked under your current rules, which catches over-broad rules before they do damage. After any change — and especially after a launch or migration — verify that your important pages are crawlable and only the intended paths are blocked. This two-minute check has saved countless sites from the silent traffic collapse that a bad robots.txt causes. Treat every robots.txt edit as a change with site-wide consequences that deserves verification, not a trivial tweak.

Crawl efficiency on large sites

On large sites, robots.txt earns its keep by steering crawlers away from the low-value URL spaces that would otherwise consume crawl budget — endless filter combinations, session URLs, and the like — so more crawling goes to the pages that matter, as our crawl budget guide discusses. Used this way, it's a genuine optimisation. But even here the discipline matters: block the low-value spaces precisely, and never so broadly that you catch important pages. On large, complex sites, a thoughtfully-maintained robots.txt is part of keeping crawling focused where it counts, which supports reliable indexing of your important content.

The bottom line

robots.txt controls which paths crawlers may request — nothing more. Its most important lesson is that blocking a page from crawling is not the same as keeping it out of the index: a blocked page can still appear in results, and a blocked page with a noindex tag can stay indexed because Google never sees the tag. To deindex, use noindex and keep the page crawlable; to manage crawl efficiency, use robots.txt. The file's site-wide reach makes mistakes uniquely expensive — a single over-broad rule can hide your whole site, most often a staging block left on after launch. Keep robots.txt minimal and deliberate, keep it consistent with your sitemap, don't block resources engines need to render, and test every change. Get it right and it's quiet housekeeping; get it wrong and it's the most damaging small file on your site.

Frequently asked questions

What does robots.txt actually do?

It's a file at the root of your domain that tells crawlers which URL paths they may or may not request — it controls crawler access, nothing more. Well-behaved crawlers read it first and respect the rules. It's useful for keeping crawlers out of low-value paths (admin areas, internal search, infinite filter combinations) that would otherwise waste crawling. It does not control what appears in search results.

Does blocking a page in robots.txt remove it from Google?

No — this is the most misunderstood point. Blocking a URL stops Google crawling it, but the URL can still appear in results (often without a description) if it's linked from elsewhere, because Google knows it exists. To keep a page out of the index, use a noindex tag and keep the page crawlable. Blocking a page you've also noindexed actually backfires: Google never crawls it, so it never sees the noindex, and the page can stay indexed.

What's the most dangerous robots.txt mistake?

A single over-broad disallow that blocks far more than intended — most catastrophically, one that blocks your whole site. It happens most often when a staging site's all-crawling block is accidentally left in place after launch, telling Google not to crawl a live site at all. Because nothing on the pages looks wrong, it can go unnoticed for weeks while traffic collapses. Always check robots.txt immediately after any launch or migration.

Should I use robots.txt to hide sensitive pages?

No. robots.txt is a public file anyone can read, so disallowing a secret path effectively advertises its location. It also doesn't reliably keep pages out of the index. For sensitive content, use proper access controls (authentication) and noindex where appropriate — not a robots.txt block, which neither hides the location nor guarantees the page stays out of search results.

Can robots.txt hurt how Google renders my pages?

Yes, if you block resources the engine needs. Blocking CSS or JavaScript files that are required to render your pages can stop Google seeing your content properly, which harms how your pages are understood and indexed. Keep robots.txt minimal and deliberate, don't block rendering resources, and test changes with the robots.txt tester before relying on them, since the file's site-wide reach makes unintended consequences unusually costly.