Robots.txt is a plain text file located at yourdomain.com/robots.txt that gives instructions to web crawlers about which parts of your site they're allowed to index. When a search bot like Googlebot visits your site, it checks this file first before crawling any pages. You write simple directives using User-agent (which bot) and Disallow or Allow (which paths to block or permit). The syntax is straightforward. A basic robots.txt might look like: User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / This tells all bots to stay out of admin and cart directories but crawl everything else. You can target specific bots by naming them, like "User-agent: Googlebot" versus "User-agent: Bingbot". You can also reference your XML sitemap location with a Sitemap directive. Common use cases include blocking duplicate content (like printer-friendly versions or session ID URLs), protecting staging environments, managing crawl budget on large sites, and keeping private sections out of search results. For ecommerce sites, we often block filter URLs and search result pages that create infinite crawl loops. Critical mistakes happen when people accidentally block their entire site with "Disallow: /" or block CSS and JavaScript files, which prevents Google from rendering pages properly. Robots.txt is also public—anyone can read it at your-domain.com/robots.txt, so never use it to hide sensitive content. Use proper authentication instead. At Ottawa SEO, we audit robots.txt files during technical site reviews because we regularly find sites blocking important sections by accident. A misconfigured file can tank your organic traffic overnight. We also see sites with no robots.txt at all, which is fine—it just means everything is crawlable by default. For most small business sites under 500 pages, you barely need one beyond blocking admin areas. But for our portfolio sites with thousands of pages, robots.txt becomes essential for directing crawl budget toward fresh content and away from archives or tag pages that add little SEO value.