Robots.txt absolutely works for legitimate crawlers. Google, Bing, Yandex, and other major search engines check your robots.txt file before crawling any URL on your domain and respect the Disallow and Allow directives you set. When configured correctly, it prevents search engines from wasting crawl budget on admin pages, duplicate content, or low-value URLs. The key limitation is that robots.txt is a request, not enforcement. Well-behaved bots follow it, but malicious scrapers, spam bots, and bad actors routinely ignore it. If you need actual security, use authentication, server-level blocks, or meta noindex tags—not robots.txt. Also, blocking a URL in robots.txt does not guarantee it won't appear in search results. If other sites link to a blocked page, Google may still index the URL with a description like "A description for this result is not available because of this site's robots.txt." Common mistakes we see in Ottawa SEO audits: - Blocking CSS or JavaScript files, which prevents Google from rendering pages properly and can hurt rankings - Using robots.txt to hide sensitive content instead of password protection or noindex tags - Forgetting the trailing wildcard, so /admin blocks one page but /admin/users.php stays open - Testing changes only in production, not using Google Search Console's robots.txt tester first When we manage a 500+ domain portfolio, robots.txt is essential for crawl budget optimization. On a small 20-page brochure site, it matters less—Google will crawl everything anyway. For large e-commerce or content sites with thousands of URLs, a clean robots.txt that blocks faceted navigation, session IDs, and internal search results can improve how efficiently Google crawls your important pages. Validate your robots.txt syntax, monitor it in Search Console for unintended blocks, and remember it's one tool in a technical SEO stack—not a security measure or a guaranteed de-indexing method.