Robots.txt can give different crawl instructions to different user agents. A group begins with a User-agent line that identifies a crawler, followed by Disallow or Allow rules for URL paths. User-agent: * addresses crawlers that do not have a more specific matching group. A Disallow rule asks a compliant crawler not to fetch matching paths. An empty Disallow value allows crawling, while Disallow: / asks that user agent not to crawl the host. Allow can identify an exception inside a broader disallowed path when the crawler supports that directive. Path matching is case-sensitive, so /Private/ and /private/ can be different URLs. The rule applies to crawling, not authorization. Malicious or non-compliant bots can ignore the file, and anyone can read it. Passwords, authentication, network controls, and server permissions are the appropriate tools for private material. Blocking a URL also does not guarantee that it disappears from search. Google can sometimes index a blocked URL based on links or other information without crawling its content. To request exclusion from search, use an index-control method that Google can crawl and read, such as a noindex meta tag or X-Robots-Tag header. Do not combine noindex with a crawl block that prevents discovery of the directive. Use narrow rules that correspond to real crawl problems, such as duplicate faceted paths or internal search results. Avoid blocking CSS or JavaScript needed to render important pages. Review each user-agent group, test patterns in a staging crawler, and confirm the live file contains only the paths you intend to restrict. For implementation, document the crawler group, the exact URL examples that should match, and the reason for each rule. Test allowed and disallowed examples with a standards-aware crawler, review the public file after deployment, and monitor server logs for unexpected access changes. This turns a broad crawl instruction into a reviewable configuration decision without treating robots.txt as security or an indexing command.