The robots.txt file lives at your site's root directory and controls what search engine bots can and cannot crawl. You create a plain text file named exactly "robots.txt" and upload it to yourdomain.com/robots.txt—no subdirectories, no variations. Basic syntax uses two main directives. "User-agent" specifies which bot the rules apply to (use * for all bots, or Googlebot, Bingbot, etc. for specific crawlers). "Disallow" tells that bot which URLs or directories to skip. For example, "User-agent: * \n Disallow: /admin/" blocks all bots from your admin folder. "Disallow: /" blocks everything, while a blank Disallow line blocks nothing. Common uses include blocking staging environments, thank-you pages, internal search results (?s= parameters), duplicate paginated content, and resource-heavy directories that waste crawl budget. You can also point crawlers to your XML sitemap with "Sitemap: https://yourdomain.com/sitemap.xml" at the bottom of the file. Key mistakes to avoid: robots.txt does not guarantee pages stay out of Google's index—it only stops crawling. If other sites link to a blocked page, Google may still index the URL without crawling it. Use noindex meta tags or X-Robots-Tag headers for true de-indexing. Also, robots.txt is public—anyone can read yourdomain.com/robots.txt, so never block directories that reveal sensitive structure unless you're also securing them server-side. At Ottawa SEO, we audit robots.txt during every technical SEO review because one misplaced slash can accidentally block your entire site. We've recovered sites losing 60–80% of their organic traffic because someone deployed "Disallow: /" instead of "Disallow: /temp/". Always test changes in Google Search Console's robots.txt Tester before pushing live, and keep the file under 500KB to ensure all crawlers read it fully. Most WordPress sites need minimal blocking—usually just /wp-admin/ and /wp-includes/, which WordPress handles by default.