Robots.txt is absolutely still used and will remain relevant as long as search engines crawl the web. Google, Bing, Yandex, and other bots check for robots.txt before crawling any site. It's the first file they request, and it tells them which URLs or sections to avoid. This prevents wasted crawl budget on admin pages, search result pages, staging environments, or duplicate content. The file lives at your root domain (example.com/robots.txt) and uses simple syntax like "Disallow: /admin/" or "Allow: /public-docs/". You can also reference your XML sitemap here with "Sitemap: https://example.com/sitemap.xml" to help crawlers find your priority pages faster. For large sites with hundreds of thousands of pages, robots.txt is critical for directing Googlebot toward revenue-generating content instead of infinite filter combinations or session IDs. Common mistakes we see: blocking CSS or JavaScript files (which Google needs to render pages properly), disallowing URLs you actually want indexed, or forgetting to remove staging blocks after launch. A misconfigured robots.txt can tank your organic traffic overnight. Use Google Search Console's robots.txt tester to validate changes before pushing live. At Ottawa SEO, we audit robots.txt on every technical SEO engagement. We've seen enterprise e-commerce sites accidentally block entire product categories and lose 40-60% of their organic visibility until we caught it. We also use robots.txt strategically to block low-value facets on our portfolio sites, preserving crawl budget for high-conversion landing pages. One nuance: robots.txt controls crawling, not indexing. If other sites link to a blocked URL, Google may still index it with limited information. To prevent indexing entirely, use noindex meta tags or X-Robots-Tag headers on the actual page. Robots.txt and meta robots directives work together, not interchangeably. Both have distinct roles in a proper technical SEO stack.