Type the domain name followed by /robots.txt directly into your browser's address bar. For instance, ottawaseo.ca/robots.txt or amazon.ca/robots.txt. The file appears as plain text with directives like User-agent, Disallow, Allow, and sometimes Sitemap references. If the site has no robots.txt, you'll see a 404 error, which is fine—it just means the site allows all crawling by default. For your own site, you create or edit robots.txt in the root directory of your web server. In WordPress that's typically public_html or www, accessible via FTP, cPanel File Manager, or your hosting control panel. Most SEO plugins like Yoast or Rank Math also let you edit robots.txt from the dashboard, though they generate a virtual file rather than writing to disk. We prefer a physical file at Ottawa SEO because it's faster to audit and version-control. Robots.txt is case-sensitive for paths but not for the filename itself. Still, always use lowercase robots.txt to avoid confusion. The file must be UTF-8 encoded plain text with Unix or Windows line breaks; fancy formatting breaks crawlers. Test your rules in Google Search Console under Index > Robots.txt Tester or the live URL inspection tool to confirm Googlebot can read it and that you haven't accidentally blocked critical pages. Common mistakes include blocking CSS, JavaScript, or image directories that Google needs to render pages, or blocking the entire /wp-admin/ in WordPress, which prevents Search Console verification. We audit robots.txt quarterly across our 500-domain portfolio because a single misplaced Disallow can tank organic traffic overnight. Keep it minimal—only block duplicate content, staging environments, admin areas, and low-value search-result pages. Everything else should remain crawlable unless you have a specific security or budget reason to restrict bots.