A robots.txt file is a plain text file you place at the root of your domain (yoursite.com/robots.txt) that gives instructions to search engine bots about which parts of your site they should or shouldn't crawl. It uses the Robots Exclusion Protocol, a standard that's been around since 1994. The main uses are controlling crawl budget, blocking low-value pages, and preventing indexation of staging environments or admin areas. If you run a 500-page site with 200 filtered or parameterized URLs that waste Googlebot's time, you can disallow those patterns so crawlers focus on pages that actually matter. You might block /admin/, /cart/, /search/, or duplicate printer-friendly versions. Common directives include "User-agent" (which bot the rule applies to), "Disallow" (what not to crawl), and "Allow" (exceptions to a disallow rule). You can also specify your XML sitemap location with "Sitemap: https://yoursite.com/sitemap.xml" so crawlers find it immediately. What robots.txt doesn't do: it doesn't prevent indexation on its own. If other sites link to a blocked URL, Google may still index it with limited information. To actually block indexation, use noindex meta tags or X-Robots-Tag headers. Also, robots.txt is a public file, so don't use it to hide sensitive content. Bots can ignore it entirely if they choose, though reputable ones like Googlebot respect it. At Ottawa SEO, we audit robots.txt early in every technical review. We've seen sites accidentally block their entire blog (Disallow: /blog/) or JavaScript resources Google needs to render pages. We've also seen ecommerce sites waste crawl budget on tens of thousands of faceted navigation URLs that should've been blocked years ago. A properly configured robots.txt file for a mid-sized site typically has 5 to 15 disallow rules targeting known problem patterns, plus the sitemap declaration. Test changes in Google Search Console's robots.txt tester before deploying, and monitor your crawl stats to confirm the impact. The file is simple but powerful when used correctly.