Your robots.txt should contain three core elements: User-agent declarations, Disallow directives for sections you actually need to block, and a Sitemap line pointing crawlers to your XML sitemap. Start with "User-agent: *" to address all bots, then add specific agents like Googlebot or Bingbot only if you need different rules for them. Disallow sparingly. Common valid blocks include /admin/, /cart/, /checkout/, duplicate parameter URLs like ?sort= or ?filter=, and staging or test directories. Many sites over-block—disallowing /wp-admin/ is fine, but blocking /wp-content/ kills your media and theme files. We see clients accidentally blocking /category/ or /tag/ pages that actually drive traffic, wasting crawl budget on a non-problem. Always include your sitemap reference at the bottom: "Sitemap: https://yourdomain.com/sitemap.xml". Google doesn't require this since you submit via Search Console, but Bing and other engines use it, and it consolidates discovery in one place. If you run multiple sitemaps (products, blog, locations), list each on its own Sitemap line. Avoid these mistakes: Never disallow your CSS or JS files—Google needs them to render pages properly since 2014. Don't block low-value pages just because they're thin; fix or noindex them instead, because robots.txt prevents indexing but also prevents Google from seeing your noindex tag. Don't use robots.txt for sensitive content—it's public and signals exactly where private areas live. Crawl-delay is non-standard and ignored by Google. Allow usually does nothing since everything is allowed by default unless disallowed. Wildcard * works in paths but not all bots respect it consistently. At Ottawa SEO, we keep most robots.txt files under 20 lines. For a 500-page site, you might block 3–5 sections. For a 50,000-page ecommerce catalog, maybe 10–15 rules covering filters, sorts, and internal search. Test changes at yoursite.com/robots.txt and validate with Google's robots.txt Tester in Search Console before deploying—one typo can deindex your entire site.