Robots.txt is a public file that tells search engine crawlers which parts of your site to avoid. It's not inherently a vulnerability, but it becomes one when site owners misuse it by listing sensitive paths they want hidden. Attackers routinely check robots.txt first because it's a roadmap to potentially interesting directories—admin panels, staging environments, backup folders, or API endpoints you've disallowed from indexing. The core problem is a misunderstanding of what robots.txt does. It's a directive for compliant crawlers, not an access control mechanism. If you disallow /admin/ in robots.txt, Google won't index it, but any human or malicious bot can still visit that URL directly. You've actually advertised its existence. Common robots.txt security mistakes include: - Disallowing /wp-admin/, /backup/, /private/, or other sensitive paths that should be protected by authentication or IP restrictions instead - Listing development or staging subdomains that weren't meant to be discovered - Exposing API routes or admin interfaces that lack proper access controls - Revealing site architecture that helps attackers understand your tech stack At Ottawa SEO, we audit robots.txt files during technical SEO reviews specifically for this issue. If a directory truly needs to be private, we implement server-level authentication, IP whitelisting, or remove it from the public web entirely. Robots.txt should only contain paths that are already public but don't need to appear in search results—like thank-you pages, internal search results, or duplicate content variations. The correct approach is defense in depth: use robots.txt to manage crawl budget and prevent indexation of low-value pages, but never rely on it for security. Sensitive resources need real authentication, proper server configuration, and regular security audits. If something shouldn't be accessed, don't just ask bots politely to stay away—actually lock the door.