Do not crawl robots txt?

Do not crawl robots txt?"Disallow" in robots.txt tells search engines not to crawl specific URLs or directories, while "noindex" (used in meta tags or HTTP headers) tells them not to index pages. A URL blocked by robots.txt can still appear in search results if other sites link to it, because Google never fetches it to see a noindex directive.AnswerThe robots.txt "Disallow" directive blocks crawlers from accessing specific paths on your site. It does not guarantee those URLs won't appear in search results—if external sites link to a disallowed URL, Google may still list it with a generic snippet like "A description for this result is not available because of this site's robots.txt." That's a critical distinction: robots.txt controls crawling, not indexing.If you want to prevent indexing entirely, use a noindex meta tag or X-Robots-Tag HTTP header on the actual page. This requires the crawler to fetch the page first, so you cannot combine noindex with a Disallow for the same URL. Google's official guidance is clear: if you block a URL in robots.txt, the bot never sees the noindex directive, and the URL may index anyway if it has inbound links.Common use cases for Disallow include admin panels (/wp-admin/), staging environments, duplicate parameter URLs (?sessionid=, ?sort=), and resource-heavy directories like /cgi-bin/. We typically disallow /wp-json/ on WordPress installs to reduce crawl budget waste, and block problematic query strings that create infinite crawl loops.Crawl budget matters more for sites above 10,000 pages or e-commerce catalogs with faceted navigation. If you have 200 pages, blocking minor directories in robots.txt is low-impact. For larger portfolios—like our 500+ domain network—strategic Disallow rules prevent Googlebot from burning time on low-value URLs and focus it on revenue pages.Watch out for accidentally blocking important resources. Google still needs to fetch CSS and JavaScript to render pages properly, so never disallow /wp-content/themes/ or /assets/. Also, if you disallow /tag/ or /author/ archives on WordPress, make sure those URLs aren't the only internal links to key content, or you'll create orphan pages.At Ottawa SEO, we audit robots.txt during every technical SEO engagement. We check for accidental blocks, compare against Search Console's coverage reports, and validate that staging subdomains are fully disallowed while production remains open. The goal is surgical control: block what wastes crawl budget, allow everything that drives organic visibility.