A robots.txt file lives at your domain root—literally yourdomain.com/robots.txt—and tells search engine crawlers which parts of your site to access or ignore. You create it as a plain text file, upload it via FTP or your CMS file manager, and structure it with User-agent lines (which bot) followed by Disallow or Allow lines (which paths). For most sites, start simple. Block admin folders, duplicate parameter URLs, and staging areas. A minimal setup looks like: User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /*?s= Sitemap: https://yourdomain.com/sitemap.xml The asterisk in User-agent means all bots. You can target specific crawlers like Googlebot or Bingbot if needed. The Disallow paths use wildcards—asterisks match any character sequence, dollar signs match line endings—so /private/$ blocks exactly /private/ but not /private/page. Common mistakes: blocking your entire site accidentally (Disallow: / with no refinement), forgetting the sitemap reference, or blocking CSS and JavaScript files Google needs to render pages. Test your file in Google Search Console under the Robots.txt Tester tool before deploying. One typo can deindex your whole site overnight. At Ottawa SEO, we treat robots.txt as crawl budget allocation, not security. It's a suggestion to well-behaved bots, not a lock. If you need real access control, use password protection or meta noindex tags. For enterprise clients with 50,000+ pages, we use robots.txt strategically—blocking faceted navigation parameters, printer-friendly versions, and internal search result pages that waste Googlebot's time. For smaller sites under 500 pages, keep it minimal: block admin, reference your sitemap, and move on. Update it when you launch new site sections or notice Google crawling junk URLs in Search Console's Crawl Stats. The file caches aggressively, so changes can take hours to propagate. Never block your sitemap itself or any URL you want indexed—sounds obvious, but we've audited sites where the previous agency did exactly that.