/robots.txt is a plain text file located at yourdomain.com/robots.txt that gives instructions to web crawlers (like Googlebot, Bingbot, or other automated bots) about which parts of your site they're allowed to crawl. It's part of the Robots Exclusion Protocol, a standard that's been around since 1994. The file uses simple directives. "User-agent" specifies which bot the rules apply to (or * for all bots), "Disallow" blocks access to specific paths, and "Allow" can override a Disallow for subdirectories. You can also include a "Sitemap" line pointing to your XML sitemap location, which helps crawlers find it faster. Common use cases include blocking crawlers from admin areas (/wp-admin/), staging environments, duplicate content sections, or resource-heavy pages that waste crawl budget. For an e-commerce site with 50,000 products, you might disallow filter URLs or session ID parameters to prevent Google from wasting time on duplicate pages. For a service business site, you might only block a handful of thank-you pages or internal search results. Critical mistakes we see regularly: blocking your entire site with "Disallow: /" (surprisingly common on new launches), blocking CSS or JavaScript files (Google needs these to render pages properly since 2015), or blocking pages you actually want indexed. A robots.txt block doesn't guarantee a page won't appear in search results if other sites link to it, it just prevents Google from crawling the content. If you need a page completely out of the index, use noindex meta tags instead. At Ottawa SEO, we audit robots.txt files in every technical SEO review because misconfigured files can tank rankings overnight. We've seen enterprise sites accidentally block their blog for months, losing 40-60% of organic traffic before anyone noticed. The file is public and accessible to anyone, so never put sensitive information in it. Test changes at google.com/webmasters/tools/robots-testing-tool before deploying, especially on large sites where a single typo can have six-figure consequences.