Robots.txt is a text file in your site's root directory that tells search engine bots which URLs they're allowed or disallowed from crawling. It exists because crawlers have limited time and resources—Google won't crawl every single page on your site during each visit, so you need to direct that crawl budget toward pages that actually matter for rankings. The main reasons to use robots.txt include blocking crawlers from admin panels, login pages, internal search result pages, thank-you pages, staging or development directories, and parameter-heavy URLs that create infinite crawl loops. For an e-commerce site with 50,000 products, you might block faceted navigation that generates millions of filter combinations, forcing Google to focus on your actual product and category pages instead. Common disallow patterns: - /wp-admin/ (WordPress backend) - /cart/ or /checkout/ (transactional pages with no SEO value) - /?s= (internal search results) - /cdn-cgi/ (Cloudflare infrastructure) - PDF files or assets that dilute crawl focus Robots.txt doesn't guarantee a page won't appear in search results—it only blocks crawling. If other sites link to a disallowed URL, Google can still index it as a result without content. To actually prevent indexing, you need a noindex meta tag or X-Robots-Tag header, which requires the page to be crawlable so Google can read that directive. At Ottawa SEO, we audit robots.txt during every technical review because blocking the wrong directory can tank your rankings overnight. We've seen sites accidentally disallow their entire blog or product catalog with one bad line. The file is case-sensitive, supports wildcards, and requires exact syntax—there's zero margin for error. For portfolio sites managing 500+ domains, we use automated monitoring to catch any robots.txt changes that could block critical sections. Always test with Google Search Console's robots.txt tester before deploying changes, and keep your directives minimal. Block what truly wastes crawl budget, not what you simply don't want ranking.