Robots.txt is important, but not in the way most site owners think. It doesn't boost rankings directly—it's a traffic control system that tells search engine crawlers which parts of your site to skip. For small sites under 500 pages, you often don't need much in robots.txt beyond blocking admin panels or duplicate parameter URLs. For larger sites or those with faceted navigation, it becomes critical for managing crawl budget so Google spends time on your money pages instead of infinite filter combinations. The biggest risk isn't having no robots.txt file—it's having a broken one. We've seen sites accidentally block their entire domain with "Disallow: /" left over from staging, killing organic traffic overnight. Googlebot respects robots.txt completely, so a typo can hide your best content. Always test changes in Google Search Console's robots.txt tester before pushing live. Common smart uses include blocking search result pages, blocking URL parameters that create duplicate content, disallowing PDF directories you want users to access but not crawlers to index separately, and rate-limiting aggressive scrapers or bad bots. You can also specify your XML sitemap location, though submitting it directly in Search Console is more reliable. At Ottawa SEO, we audit robots.txt on every technical review. For e-commerce clients, we typically see 20–40% of crawl budget wasted on faceted URLs that should be disallowed. For publishing sites with thousands of tag pages, blocking low-value taxonomy archives lets Google focus on articles. One Toronto client had accidentally blocked their blog subdirectory for eight months—traffic doubled within three weeks of fixing it. Robots.txt won't remove pages already indexed. For that, you need noindex meta tags or URL removal requests. Think of robots.txt as a "don't waste time here" signal, not a "remove this from Google" command. Most sites need one, but keep it simple—five to ten directives cover 95% of real use cases.