Site owners use robots.txt when a crawler does not need to fetch every URL the server can generate. Common examples include internal search results, duplicate filter combinations, session or tracking parameters, and resource-heavy endpoints that have no useful search landing-page role. This can matter on large or technically complex sites where many URL combinations compete for server and crawler attention. The goal is not to “send authority” to other pages or force Google to index preferred URLs. It is simply to reduce crawling of clearly unnecessary patterns while leaving important pages and rendering resources accessible. The file can also publish Sitemap lines so crawlers can discover XML sitemap locations. Sitemaps and robots.txt remain separate controls: a sitemap suggests URLs for discovery, while robots.txt can restrict fetching. Listing a URL in a sitemap does not override a matching crawl restriction. Do not use robots.txt for confidential information, login protection, or reliable search removal. The file is public and non-compliant bots can ignore it. A blocked URL can sometimes appear in search based on links even though its content was not crawled. Authentication protects private resources; a crawlable noindex directive or an appropriate removal tool addresses search exclusion. Use the smallest rule set that solves a verified crawl problem. Document why each broad pattern exists, review changes with developers, test representative URLs in a staging crawler, and monitor the live file after releases. Crawl management is the reason to use robots.txt; ranking promises and security are not. For implementation, document the crawler group, the exact URL examples that should match, and the reason for each rule. Test allowed and disallowed examples with a standards-aware crawler, review the public file after deployment, and monitor server logs for unexpected access changes. This turns a broad crawl instruction into a reviewable configuration decision without treating robots.txt as security or an indexing command.