Robots.txt is a text file you place at yoursite.com/robots.txt that instructs search engine crawlers which URLs they're allowed to request. It uses the Robots Exclusion Protocol, a standard most reputable bots respect. When Googlebot or Bingbot hits your site, they check this file first before crawling anything else. The file uses simple directives. User-agent specifies which bot the rule applies to (User-agent: * means all bots, User-agent: Googlebot targets only Google). Disallow tells bots not to crawl certain paths. Allow can override a Disallow for subdirectories. You can also reference your XML sitemap location here with Sitemap: https://yoursite.com/sitemap.xml. Common uses include blocking admin areas (/wp-admin/), thank-you pages that shouldn't rank, duplicate parameter URLs (?sort=price), and PDF directories you don't want indexed. For e-commerce sites, blocking faceted navigation parameters can save hundreds of thousands of wasted crawl requests monthly. We've seen crawl budget improvements of 40–60% on large sites after cleaning up robots.txt to block low-value paths. Critical mistakes to avoid: Never disallow pages you want indexed. Robots.txt blocks crawling, not indexing. If a blocked URL gets links, Google can still index it based on external signals, showing it in results with no description. If you need to prevent indexing, use noindex meta tags or X-Robots-Tag headers instead. Also, robots.txt is public—anyone can read yoursite.com/robots.txt, so don't list sensitive directories you're trying to hide. At Ottawa SEO, we audit robots.txt in every technical review. For our portfolio sites, we typically block search parameters, staging subdirectories, and internal search results while ensuring all valuable content paths remain open. We also validate the syntax with Google Search Console's robots.txt tester, because one typo can accidentally block your entire site. Most SMB sites need 10–20 lines maximum. If yours is 200+ lines, you're likely overcomplicating it or should handle exclusions at the meta-tag level instead.