A robots.txt file sits at your domain root and uses plain text directives to control crawler access. The basic structure includes a User-agent line specifying which bot the rules apply to, followed by Disallow or Allow lines with URL paths. A minimal example: "User-agent: *" targets all bots, "Disallow: /admin/" blocks the admin folder, and "Allow: /" permits everything else. You can also reference your XML sitemap with "Sitemap: https://yoursite.com/sitemap.xml". Most production files look more complex. You might block staging environments, thank-you pages, internal search results, or duplicate parameter-heavy URLs. For example: "User-agent: *", "Disallow: /checkout/", "Disallow: /*?utm_source=", "Disallow: /search/", "Allow: /checkout/success". The Allow directive overrides broader Disallow rules when paths overlap. Googlebot and Bingbot respect these directives; some scrapers ignore them entirely. Common mistakes we see: blocking CSS or JavaScript files (kills Google's rendering), using robots.txt when you meant robots meta tags (robots.txt controls crawling, meta tags control indexing), or forgetting the wildcard matters. "Disallow: /private" blocks /private-page but also /privatepage, while "Disallow: /private/" only blocks the folder. Case sensitivity depends on your server, so be consistent. Ottawa SEO runs robots.txt files that typically sit under 30 lines unless the site has complex faceted navigation or staging subdomains. We reference the sitemap, block obvious junk like /cgi-bin/ or /wp-admin/, and allow everything else by default. For ecommerce clients, we often disallow filter parameters and user-generated review endpoints that create infinite crawl loops. For portfolio sites across our 500-domain network, we keep them minimal since over-blocking causes more problems than under-blocking. Test your file at yoursite.com/robots.txt and use Google Search Console's robots.txt tester before deploying. One misplaced slash can deindex your entire site, and the fix takes weeks to propagate. Keep a changelog if multiple people touch it, and never block resources required for rendering unless you know exactly why.