When you add a "Disallow" line in your robots.txt file, you're instructing search engine crawlers not to request or crawl the URLs that match that pattern. For example, "Disallow: /admin/" tells bots to skip everything in your admin directory. "Disallow: /" blocks the entire site, while an empty "Disallow:" line actually allows everything. Robots.txt is a plain text file placed at your domain root (yourdomain.com/robots.txt). Crawlers like Googlebot check this file before crawling your site. The disallow directive doesn't guarantee pages won't get indexed—if other sites link to a blocked URL, Google may still index it based on external signals, just without crawling the content. To prevent indexing entirely, you need a noindex meta tag or X-Robots-Tag header on the actual page. Common uses include blocking staging environments, admin panels, internal search result pages, duplicate filtered pages on ecommerce sites, and resource-heavy directories that waste crawl budget. We frequently disallow /wp-admin/, /cart/, and parameterized URLs like /?sort= on client sites to keep crawlers focused on indexable content. Syntax matters. "Disallow: /private" blocks /private, /private.html, and /private/page, but "Disallow: /private/" only blocks URLs starting with /private/ (with the trailing slash). Wildcards (*) and end-of-path ($) are supported by Google and Bing but not universally, so test in Google Search Console's robots.txt tester. Mistakes we see constantly: accidentally disallowing CSS or JavaScript files, which prevents Google from rendering pages properly, or blocking entire site sections that should be indexed. Always verify your robots.txt logic before pushing live. At Ottawa SEO, we audit robots.txt during every technical SEO engagement because one wrong line can tank a site's crawlability. It's a blunt instrument—powerful when used correctly, catastrophic when misconfigured.