Where is a robots.txt file located?

Where is a robots.txt file located?Robots.txt belongs at /robots.txt on the root of each protocol and host it governs, such as https://example.com/robots.txt. A file in a subdirectory has no robots.txt authority, and subdomains need their own file when they require separate rules.AnswerRobots.txt is located at the top-level /robots.txt path for the protocol, host, and port it governs. For https://www.example.com, the expected location is https://www.example.com/robots.txt. A copy at /docs/robots.txt or /public/robots.txt does not control crawling for those directories.Scope follows the origin. The file for https://example.com does not automatically govern https://shop.example.com, and the HTTPS file is separate from the HTTP origin. A subdomain that needs different rules should expose its own /robots.txt file. Sites using non-standard ports should also consider port scope when testing.The URL must be publicly fetchable by the crawler you expect to follow it. Server status codes, redirect chains, timeouts, and caching can affect how a crawler interprets availability, and behavior varies by crawler and failure type. Avoid absolute claims that every crawler handles redirects or errors identically; consult the crawler's current documentation and keep the root URL stable.After deployment, request the exact /robots.txt URL directly, confirm the response body and content type are sensible, and inspect server logs or a standards-aware crawler. Check both www and non-www hosts, relevant subdomains, and HTTP-to-HTTPS behavior. Search Console's former robots.txt tester is retired, so it should not be part of current instructions.Location errors are usually simple: the file was uploaded beneath the web root, only one subdomain received it, or a proxy serves unexpected content at the root URL. Testing the exact public origin reveals those mistakes without making unsupported assumptions about indexing.For implementation, document the crawler group, the exact URL examples that should match, and the reason for each rule. Test allowed and disallowed examples with a standards-aware crawler, review the public file after deployment, and monitor server logs for unexpected access changes. This turns a broad crawl instruction into a reviewable configuration decision without treating robots.txt as security or an indexing command.