Robots.txt is a plain-text file that lives at the root of your domain and gives search engine bots instructions about which parts of your site they're allowed to crawl. It's the first thing most crawlers check when they visit your site, so it acts as a gatekeeper for your crawl budget and controls what gets indexed. The syntax is straightforward. You specify a User-agent (the bot you're targeting, like Googlebot or Bingbot, or * for all bots), then use Disallow to block specific paths or file types, and Allow to permit exceptions within blocked sections. For example, "User-agent: *" followed by "Disallow: /admin/" tells all bots to stay out of your admin directory. You can also reference your XML sitemap with a Sitemap directive to help crawlers find your important pages faster. Common use cases include blocking staging environments, thank-you pages, internal search results, duplicate filtered URLs, and resource-heavy pages that waste crawl budget without adding SEO value. You might block /wp-admin/ on WordPress sites or /checkout/ on e-commerce stores. Just never block CSS, JavaScript, or images that Google needs to render pages properly—that's been a ranking factor since 2015. At Ottawa SEO, we treat robots.txt as a scalpel, not a sledgehammer. Mistakes here can tank your rankings overnight if you accidentally disallow your entire site (we've seen "Disallow: /" wipe out indexing for clients who inherited bad configs). Always test changes in Google Search Console's robots.txt Tester before pushing live, and remember that robots.txt is a request, not enforcement—malicious bots ignore it, and it doesn't hide sensitive data from determined users. One nuance: if a URL is disallowed in robots.txt but other sites link to it, Google may still index the URL without crawling it, showing the link and title but no snippet. If you truly need to block indexing, combine robots.txt blocks with noindex meta tags on the actual pages, or use server-level authentication for genuinely sensitive areas.