robots.txt controls crawling, not indexing. This is one of the most misunderstood concepts in technical SEO. If you block a page in robots.txt, Google won't fetch its content, but the URL can still appear in search results as an indexed entry with a snippet like "A description for this result is not available because of this site's robots.txt." This happens when external sites link to the blocked URL. To actually prevent indexing, you need a noindex directive—either a meta robots tag in the HTML head or an X-Robots-Tag in the HTTP response header. The catch: Googlebot must be able to crawl the page to see the noindex instruction. If you block crawling via robots.txt and add noindex, Google can't read the noindex tag, so the URL may remain indexed. Common scenarios we see at Ottawa SEO: - Staging sites blocked by robots.txt but still indexed because developers linked to them from production - Duplicate parameter URLs disallowed in robots.txt that pile up in Google's index from external referrers - Admin or login pages blocked via robots.txt showing up in SERPs with no description The correct approach depends on your goal. For true de-indexing, allow crawling and apply noindex. For sensitive content you don't want Google to cache, use noindex plus authentication at the server level. For low-value pages you want crawl budget preserved on, canonical tags or parameter handling in Search Console often work better than robots.txt blocks. One exception: if a page has already been indexed and you only add robots.txt blocking without noindex, Google will likely keep it indexed indefinitely because it can't re-crawl to discover the noindex. We've seen URLs linger in the index for 18+ months this way. Always verify your approach in Search Console's URL Inspection tool and check the actual indexed result, not just your local testing environment.