Robots.txt is a plain text file that lives at your domain root and gives crawl instructions to bots. You create it manually or through your CMS, upload it to the top level of your site, and use two core directives: User-agent (which bot the rule applies to) and Disallow or Allow (which paths to block or permit). A basic setup looks like "User-agent: *" (all bots) followed by "Disallow: /admin/" to block your admin folder. If you want to block nothing, use "Disallow:" with no path, or just omit the file entirely. Common uses include blocking duplicate staging environments, hiding thank-you pages that don't need organic traffic, preventing crawlers from wasting budget on search result pages or filtered product views, and keeping private directories out of indexes. You can also reference your XML sitemap with "Sitemap: https://yourdomain.com/sitemap.xml" so crawlers find it immediately. Critical mistakes we see: accidentally blocking CSS or JavaScript files (older Google advice said to block them, now it breaks rendering), disallowing your entire site with "Disallow: /" when you meant to block one folder, or assuming robots.txt guarantees privacy (it doesn't, it's a request, not a firewall, and the file itself is public). If a page is already indexed, blocking it in robots.txt won't remove it; you need noindex tags or Search Console removal requests for that. Test your file with Google Search Console's robots.txt tester before going live. For most small to mid-size sites, a minimal robots.txt blocking only admin, cart, and checkout paths works fine. Larger sites or those with heavy crawl activity should block paginated archives, low-value filters, and internal search to preserve crawl budget for pages that actually earn traffic. At Ottawa SEO, we audit robots.txt during technical reviews because one wrong slash can accidentally deindex revenue pages, and we've recovered sites where the previous dev blocked the entire domain by mistake.