Navigate to yoursite.com/robots.txt in any browser to see your file. If the file exists, you'll see plain-text directives like User-agent and Disallow lines. If you get a 404 error, no robots.txt is present, which means crawlers have unrestricted access by default. Once you view the file, verify these elements: User-agent lines specify which bots the rules apply to (Googlebot, Bingbot, or * for all). Disallow lines block specific paths or patterns. Allow lines can override Disallow rules for Googlebot. Sitemap entries point crawlers to your XML sitemap location. Watch for accidental blocks like "Disallow: /" which stops all crawling, or leftover staging rules like "Disallow: /wp-admin" paired with overly broad patterns that block critical pages. Google Search Console provides a robots.txt tester under the old interface (some accounts still have it) or you can test URLs against your robots.txt using the URL Inspection tool to see if Googlebot can access them. Screaming Frog and similar crawlers will also flag robots.txt issues during site audits. For multi-domain portfolios like Ottawa SEO's 500+ sites, we automate checks with scripts that curl each /robots.txt endpoint and grep for dangerous patterns monthly. Common mistakes we catch: blocking CSS or JavaScript files that Google needs to render pages properly, forgetting to remove "Disallow: /" after launch when a site was previously in development, and conflicting directives where Allow and Disallow overlap without clear precedence. The order matters for Google but not all bots interpret rules identically. If you're managing client sites, document what's blocked and why. A robots.txt that blocks /cart/ might make sense for a blog but kills an e-commerce site's indexing. Always cross-reference with your sitemap, canonical tags, and noindex directives since robots.txt is just one access control layer. It doesn't remove already-indexed pages; you need noindex meta tags or 410 status codes for that.