Robots.txt is a plain-text file placed at the root of a domain (yoursite.com/robots.txt) that tells search-engine crawlers which URLs they may or may not request. Most bots check it first, and it shapes crawl behaviour across a site — but it is a polite instruction, not an enforced lock.
Robots.txt governs crawling, not indexing, and confusing the two causes real damage. Disallowing a URL stops compliant crawlers from fetching it, yet if that URL is linked from elsewhere Google can still list it — usually with no description, since the page was never read. To keep something out of results you must let it be crawled and add a noindex tag, or put it behind a login. Blocking the URL in robots.txt actually prevents Google from ever seeing that noindex.
Rules are grouped by user-agent, and a basic file reads plainly:
During a redesign, a developer left the staging rule "Disallow: /" in the file when pushing to production, blocking the entire site from crawling. Over the following two weeks pages dropped out of Google's index and organic traffic collapsed. The fix was one line; recovery took a month. Always review robots.txt at launch, test it in Search Console, and never rely on it to hide sensitive URLs — the file is public, so anyone can read exactly what you disallow. Getting these details right is foundational technical SEO.