Robots.txt is absolutely still used and continues to be a foundational technical SEO tool in 2025. Every major search engine—Google, Bing, Yandex, Baidu—checks robots.txt before crawling your site. It's the first file crawlers request, and compliance is built into their protocols. The primary use cases haven't changed: blocking crawlers from staging environments, admin panels, search result pages, duplicate content variations, and resource-heavy directories that waste crawl budget. For a 500+ domain portfolio like ours at Ottawa SEO, robots.txt prevents Google from indexing thousands of development URLs, filtered product pages, and internal search results that would dilute crawl efficiency. What has evolved is how we layer it with other directives. Robots.txt blocks crawling but doesn't guarantee de-indexing—URLs can still appear in search results if linked externally. For true exclusion, you need noindex meta tags or X-Robots-Tag headers on the actual pages. We typically use robots.txt for broad category blocks (like /wp-admin/) and noindex for specific pages that need guaranteed exclusion. Crawl budget matters more than most people think. A Toronto e-commerce client with 40,000 SKUs was wasting 60% of their crawl budget on faceted navigation URLs until we tightened their robots.txt rules. Within three weeks, Google started crawling product pages 2.3× more frequently. Common mistakes to avoid: never block CSS or JavaScript files (Google needs these to render pages properly), don't use robots.txt to hide sensitive data (it's publicly viewable at yoursite.com/robots.txt), and remember that disallowing a URL in robots.txt while also trying to noindex it creates a conflict—Google can't see the noindex tag if it's blocked from crawling. The syntax is simple but unforgiving. One typo in a wildcard pattern can accidentally block your entire site. We always test changes in Google Search Console's robots.txt tester before deploying, and we monitor crawl stats for unexpected drops that signal blocking issues.