Type the domain followed by /robots.txt directly in your browser's address bar. For example, ottawaseo.com/robots.txt or amazon.ca/robots.txt. The file displays immediately if it exists, showing which user-agents can access which paths. The robots.txt file lives at the root level of every domain. It's a plain text file that tells search engine crawlers which URLs or directories to avoid. If no file exists, you'll see a 404 error, which means the site allows all bots to crawl everything by default. When reviewing a robots.txt file, look for these key elements: - User-agent lines (which bots the rules apply to, like Googlebot or Bingbot) - Disallow directives (paths blocked from crawling) - Allow directives (exceptions to disallow rules) - Sitemap location (often listed at the bottom) - Crawl-delay settings (rare on modern sites, more common for aggressive scrapers) Common mistakes we catch during technical audits include blocking CSS or JavaScript resources that Google needs to render pages, disallowing entire staging environments that leaked into production robots.txt, or accidentally blocking critical conversion pages. We've seen e-commerce sites block /cart/ or SaaS companies block /pricing/ because someone copied a template without understanding the syntax. For competitive analysis, checking a competitor's robots.txt reveals their site structure, which CMS they use (via blocked admin paths), and whether they're hiding sections from search engines intentionally. You might find disallowed /blog-staging/ paths or blocked faceted navigation parameters that show how they handle duplicate content. Google Search Console offers a robots.txt tester under the old version of the tool, though it's being phased out. For bulk checks across your domain portfolio, command-line tools like curl or wget work well. At Ottawa SEO, we monitor robots.txt changes across our 500+ domains using scheduled scripts that alert us if critical paths get accidentally blocked, since one wrong line can tank organic traffic overnight.