Google absolutely respects robots.txt and has publicly confirmed this for decades. When you disallow a path in robots.txt, Googlebot will not crawl those pages. This is verifiable in your server logs—you won't see Googlebot requests for blocked URLs under normal circumstances. The critical nuance: blocking crawling doesn't guarantee blocking indexing. If other sites link to a disallowed URL, Google may still list it in search results with a generic snippet like "A description for this result is not available because of this site's robots.txt." You'll see the URL and title (pulled from anchor text), but no crawled content. To prevent indexing entirely, you need a noindex meta tag or X-Robots-Tag header, which requires allowing the crawl so Google can read the directive. Robots.txt is also completely public—anyone can view yoursite.com/robots.txt. Blocking admin or staging areas in robots.txt actually advertises their existence to competitors and bad actors. For genuinely sensitive content, use authentication or noindex instead. Common robots.txt mistakes we see at Ottawa SEO: - Blocking CSS or JavaScript files, which prevents Google from rendering pages properly and can harm rankings - Disallowing pagination or filter URLs that should be crawlable for faceted navigation sites - Using robots.txt for duplicate content management instead of canonicals - Forgetting that different bots read different directives—blocking Googlebot doesn't block Bingbot or others Google also crawls robots.txt itself regularly to check for updates, usually every 24 hours but sometimes faster for high-authority sites. Changes aren't instant—existing crawl queues may still process for days. For our portfolio sites, we use robots.txt exclusively for true crawl-budget optimization on large sites or blocking resource-heavy admin areas where crawling wastes server resources. For content we want to keep out of the index, we use noindex tags and allow the crawl so the directive is read. Robots.txt is a blunt instrument—precise when used correctly, but it won't save you from indexing issues alone.