Robots.txt is a publicly accessible file (yoursite.com/robots.txt) that gives instructions to search engine crawlers about which parts of your site they should or shouldn't crawl. It uses a simple syntax with User-agent lines to target specific bots and Disallow or Allow directives to block or permit access to URL paths. The primary use cases are blocking crawlers from admin panels, staging directories, internal search result pages, duplicate content created by URL parameters, and resource-heavy sections that provide no SEO value. For example, you might block /wp-admin/ on WordPress sites or /cart/ on e-commerce platforms where crawling session-specific pages wastes your crawl budget. You can also point crawlers to your XML sitemap using a Sitemap directive. Robots.txt does not make pages disappear from search results. If a blocked page has inbound links, Google may still index the URL with a generic description. To actually prevent indexing, you need a noindex meta tag or X-Robots-Tag header. This is a common misconception that trips up site owners who expect robots.txt to remove pages from SERPs. At Ottawa SEO, we audit robots.txt files during technical reviews because misconfigurations are surprisingly common. We've seen sites accidentally block their entire blog with a single incorrect directive or forget to remove a Disallow after migrating from staging. One slash in the wrong place can block thousands of pages from Google. Key points for implementation: keep it simple, test changes with Google Search Console's robots.txt Tester before deploying, and remember that malicious bots often ignore it entirely since it's just a suggestion protocol, not a security mechanism. For crawl budget management on large sites (10,000+ pages), strategic use of robots.txt combined with proper internal linking and sitemap hygiene makes a measurable difference in how efficiently Google discovers your important content. Most small business sites under 500 pages don't need aggressive robots.txt rules beyond blocking admin areas.