A robots.txt file is a plain text document that lives at yourdomain.com/robots.txt and controls how search engine crawlers interact with your site. When a bot like Googlebot visits your site, it checks this file first to see which URLs it's allowed to crawl and which are off-limits. It's not a security measure—anyone can read your robots.txt—but it's essential for managing crawl budget and preventing indexation of low-value pages. The file uses simple directives. User-agent specifies which bot the rule applies to (like Googlebot, Bingbot, or * for all bots). Disallow tells bots not to crawl specific paths. Allow can override a Disallow for subdirectories. You can also include a Sitemap directive pointing to your XML sitemap, which helps crawlers find your important pages faster. Common uses include blocking admin areas, duplicate content from URL parameters, staging environments, and thank-you pages. For a 500+ domain portfolio like ours, robots.txt is critical for controlling which regional or test subdomains get crawled. We've seen cases where a misconfigured robots.txt blocked an entire site from Google for weeks, tanking traffic overnight, so syntax matters. Key mistakes to avoid: - Blocking CSS or JavaScript files, which prevents Google from rendering pages properly - Using robots.txt to hide sensitive data (it doesn't block access, just crawling) - Forgetting that Disallow doesn't guarantee de-indexation—already indexed pages may stay in results - Over-blocking and starving Google of crawl paths to important content At Ottawa SEO, we treat robots.txt as a scalpel, not a hammer. For most small to mid-sized sites, a minimal file that blocks admin paths and points to the sitemap is enough. Larger sites need strategic crawl budget allocation, especially if you're dealing with faceted navigation or thousands of product pages. Always test changes in Google Search Console's robots.txt tester before deploying, and monitor crawl stats after updates to catch issues early.