A robots.txt file is a text document you place at yourdomain.com/robots.txt that gives instructions to search engine bots about which parts of your site they're allowed to crawl. Every legitimate crawler checks this file before requesting other pages, so it acts as your first line of control over how search engines interact with your site. The file uses simple directives. User-agent specifies which bot the rule applies to (Googlebot, Bingbot, or * for all). Disallow tells bots not to access certain paths. Allow can override a disallow for specific subfolders. You can also reference your XML sitemap location here, which helps crawlers find your priority pages faster. Common use cases include blocking admin areas (/wp-admin/), filtering parameters that create duplicate content (?sort=, ?page=), preventing indexation of staging environments, and conserving crawl budget on large sites by keeping bots away from low-value sections like internal search results or printer-friendly versions. For a 500-page site, this might not matter much. For a 50,000-page e-commerce or news site, managing crawl budget through robots.txt can directly impact how fresh your important pages appear in search results. Critical mistakes we see: disallowing CSS or JavaScript files, which prevents Google from rendering pages properly; blocking entire sections you actually want indexed (use meta noindex tags on the pages instead); forgetting that robots.txt is a public file anyone can read at yourdomain.com/robots.txt, so never put sensitive URLs there. Also, robots.txt is a directive, not enforcement. Malicious bots ignore it, and Google may still index a blocked URL if it finds links pointing to it, though it won't crawl the content. At Ottawa SEO, we audit robots.txt in every technical review because one wrong line can deindex an entire site. We've inherited clients who accidentally disallowed their whole domain for months. Test changes in Google Search Console's robots.txt tester before deploying, and keep the file minimal—only block what actively harms your crawl efficiency or creates indexation problems.