You need robots.txt when you have pages you don't want search engines wasting time crawling. It's not about hiding secrets—it's about crawl budget efficiency and preventing indexation of junk that dilutes your site's perceived quality. Block admin and utility pages first. WordPress /wp-admin/, Shopify /admin/, staging subdomains, development environments, thank-you pages, and internal search result URLs (?s=, /search/) are prime candidates. Google doesn't need to see your login screen or crawl thousands of filtered product views that create no unique value. Use it to manage crawl budget on large sites. If you run 500+ pages, every bot hit costs server resources and counts against how much Google will crawl per day. Block PDF directories if you have hundreds of downloads, printer-friendly versions of pages, or parameter-heavy faceted navigation (/products?color=red&size=M&sort=price). E-commerce sites commonly block cart and checkout flows. Common robots.txt entries we deploy: - Disallow: /wp-admin/ (except /wp-admin/admin-ajax.php for WordPress) - Disallow: /*?* (blocks URL parameters, but use carefully—can block legitimate paginated content) - Disallow: /cgi-bin/ - Disallow: /staging/ Don't use robots.txt to hide sensitive content. It's a publicly readable file at yourdomain.com/robots.txt, and blocking a URL doesn't remove it from Google if it's already indexed or linked externally. For actual removal, use noindex meta tags or password protection. At Ottawa SEO, we audit robots.txt during every technical review. We've seen sites accidentally block their entire blog (/blog/) or block CSS/JS files, which prevents Google from rendering pages properly and kills rankings. Test changes with Google Search Console's robots.txt Tester before deploying, and always include your XML sitemap location at the bottom (Sitemap: https://yourdomain.com/sitemap.xml) to help crawlers find your priority content faster.