Robots.txt tells search engine crawlers which URLs or directories they should skip when crawling your site. The primary reason to use it is crawl budget optimization: every site gets a finite number of pages Google will crawl per day, so you want to steer crawlers toward your valuable, indexable content instead of letting them burn resources on admin login pages, internal search results, PDF archives, or duplicate parameter-based URLs. Common use cases include blocking /wp-admin/ and /wp-includes/ on WordPress sites, excluding faceted navigation that creates thousands of near-duplicate product pages, or preventing crawlers from hitting resource-heavy pages like infinite-scroll APIs or PDF generators that slow down your server. You can also block entire user-agents—like notorious scrapers—though sophisticated bots often ignore robots.txt anyway. Robots.txt is not a security measure. Blocked URLs can still appear in search results if other sites link to them, and anyone can read your robots.txt file by visiting yourdomain.com/robots.txt. If you truly need to hide a page, use password protection or noindex meta tags plus authentication. Robots.txt is just a polite request; compliant crawlers respect it, but malicious ones don't. When we audit client sites at Ottawa SEO, we often find over-blocking—people accidentally disallow their entire site or critical CSS/JS files, which tanks rendering and rankings. Google Search Console shows crawl stats and blocked resources, so check that regularly. We also see the opposite: sites with no robots.txt that let crawlers waste time on thousands of tag pages or search filters. The sweet spot is a lean robots.txt that blocks known waste while allowing everything indexable, paired with XML sitemaps to guide crawlers toward priority pages. For most small-to-medium sites, a 10-line robots.txt covering admin paths and a few problem directories is enough. Larger ecommerce or news sites might need dynamic rules or multiple sitemaps referenced in the file.