Access any website's robots.txt by typing the domain followed by /robots.txt directly into your browser—like ottawaseo.com/robots.txt or amazon.ca/robots.txt. The file must live at the root domain level to be recognized by crawlers, so subdomains and subdirectories won't work. For your own WordPress site, you can view and edit robots.txt through Yoast SEO or Rank Math under their tools section, or access it via FTP in the public_html root folder. On custom builds, check the web server's document root—typically /var/www/html on Linux or the equivalent on your hosting setup. If you're on Shopify or Squarespace, the platform auto-generates robots.txt and you'll usually find limited editing options in the SEO settings panel. When auditing client sites at Ottawa SEO, we check robots.txt in three scenarios: initial technical audit to confirm we're not blocking important URLs, after a dev push to catch accidental disallow rules, and when diagnosing sudden indexation drops in Search Console. Google Search Console also surfaces robots.txt errors under the Coverage report if directives conflict with your sitemap or block Googlebot. Common mistakes to watch for include leftover staging directives (Disallow: / from a dev environment), blocking CSS or JavaScript files that Google needs to render pages, or conflicting rules where one line allows and another blocks the same path. The order matters—most specific rules should come first, and crawlers read top to bottom until they hit a match. If you don't see a robots.txt file when you visit /robots.txt, the site simply doesn't have one, which means crawlers treat everything as allowed by default. That's fine for many small sites, but once you're managing a portfolio with hundreds of domains, explicit robots.txt control prevents budget waste on irrelevant URLs like /cart, /checkout, or paginated archives. Always validate syntax with Google's robots.txt tester in Search Console before deploying changes.