robots.txt is a publicly accessible file (yourdomain.com/robots.txt) that gives instructions to search engine bots about which parts of your site they're allowed to crawl. It uses a simple syntax with User-agent declarations (which bot) and Disallow or Allow directives (which paths). For example, "Disallow: /admin/" tells all bots to skip your admin folder. The file controls crawl access, not indexing. If you block /private-page/ in robots.txt but someone links to it externally, Google may still index the URL with a descriptor like "A description for this result is not available because of this site's robots.txt." To truly prevent indexing, use noindex meta tags or X-Robots-Tag headers. Common uses include blocking duplicate staging environments, protecting resource-heavy pages from excessive crawling, hiding low-value sections like /cart/ or /checkout/, and specifying your sitemap location with a Sitemap directive. Many sites disallow /wp-admin/ and /wp-includes/ on WordPress to reduce wasted crawl budget, though Google ignores blocks on CSS and JS because it needs those to render pages properly. Mistakes are costly. Accidentally blocking your entire site with "Disallow: /" is more common than you'd think, especially after a staging-to-production migration. We've seen e-commerce sites tank overnight because a developer forgot to swap out the restrictive robots.txt. Always validate syntax with Google Search Console's robots.txt tester before deploying. At Ottawa SEO, we audit robots.txt early in every technical review. We check for accidental blocks on important sections, confirm the sitemap reference is present and correct, and ensure directives align with crawl budget strategy. For most sites under 10,000 pages, robots.txt is minimal—block admin areas, reference the sitemap, done. Larger portfolios need more nuance to guide bots toward fresh content and away from infinite filtering facets. The file is simple but unforgiving, so treat every edit like pushing code to production.