A robots.txt file sits at yoursite.com/robots.txt and gives instructions to search engine bots about what they're allowed to crawl. The primary uses are controlling crawl budget on large sites, blocking low-value sections like admin panels or staging directories, and preventing indexation of duplicate content or parameter-heavy URLs that waste resources. The file uses simple directives. User-agent specifies which bot the rule applies to (Googlebot, Bingbot, or * for all). Disallow blocks specific paths. Allow overrides a disallow for a subfolder. You can also reference your XML sitemap location here, which most crawlers will pick up. Common mistakes we see: blocking CSS or JavaScript files, which prevents Google from rendering pages properly and can hurt rankings. Disallowing your entire site by accident (Disallow: / under User-agent: *) during a site migration. Treating robots.txt as a security tool when it's purely a crawling hint, any blocked URL can still appear in search results if linked externally, just without a snippet. For most small to mid-sized sites under 10,000 pages, you don't need aggressive robots.txt rules. Google's crawl budget is generous. We typically use it to block /wp-admin/, /cgi-bin/, thank-you pages, or internal search result pages that generate infinite URL variations. On larger portfolio sites or ecommerce platforms with hundreds of thousands of SKUs, blocking faceted navigation parameters (?color=, ?sort=) saves crawl budget for pages that actually matter. Test changes in Google Search Console's robots.txt Tester before deploying. A single typo can accidentally block your entire site. The file is case-sensitive and follows specific syntax, so Disallow: /Admin/ won't block /admin/. For Ottawa SEO's portfolio, we keep robots.txt lean, block only what genuinely shouldn't be crawled, and rely on noindex meta tags when we want a page accessible but not indexed.