Your robots.txt file must live at the root of your domain—directly accessible at yourdomain.com/robots.txt. This isn't optional. Google, Bing, and every other legitimate crawler expect it there and won't check subdirectories or alternate locations. If you're running a subdomain like blog.yourdomain.com, it needs its own robots.txt at blog.yourdomain.com/robots.txt. For most sites, this means uploading the file via FTP or your hosting control panel to the public_html, www, or httpdocs folder, depending on your server setup. On WordPress, that's typically the same directory where you find wp-config.php. On platforms like Shopify or Wix, you often access it through their built-in SEO settings rather than direct file upload. A few critical rules: the file must be named exactly "robots.txt" in lowercase, use UTF-8 encoding, and stay under 500 KB (though anything over 50 KB suggests you're doing something questionable). If you're using HTTPS—which you should be—serve robots.txt over HTTPS as well. Mixed protocol setups confuse crawlers and waste your crawl budget. Common mistakes we see at Ottawa SEO: clients accidentally block their entire site with a staging-environment robots.txt that survived launch, or they create separate files for different sections and wonder why only the root one works. Also, don't rely on robots.txt for sensitive content. It's a request, not security. Disallowed pages can still get indexed if other sites link to them, and the URLs remain visible in the file itself. Test your setup at yourdomain.com/robots.txt in a browser. If it doesn't load, neither will Googlebot see it. Use Google Search Console's robots.txt tester to verify syntax, and check that your User-agent directives match what you actually want to control. We typically keep ours minimal—blocking admin areas, duplicate parameter URLs, and internal search results—then handle everything else through noindex tags and canonical URLs where we need precision.