A robots.txt file is a plain text file that sits at your domain root and tells search engine crawlers which parts of your site they can access. Creating one takes under five minutes. Open any text editor, save a file named exactly 'robots.txt' (lowercase, no .html or other extension), and upload it to your root directory so it's accessible at https://yourdomain.com/robots.txt. The basic syntax uses two components: User-agent (which bot you're addressing) and Disallow or Allow (which paths to block or permit). For a wide-open site, you'd write 'User-agent: *' on one line, then 'Disallow:' on the next with nothing after the colon. That tells all bots they can crawl everything. To block a folder, use 'Disallow: /admin/' or 'Disallow: /wp-admin/'. To block a specific bot, replace the asterisk with its name, like 'User-agent: AhrefsBot'. Most sites also include a Sitemap directive at the bottom: 'Sitemap: https://yourdomain.com/sitemap.xml'. This doesn't control crawling but helps bots find your XML sitemap faster. You can list multiple sitemaps if you have separate ones for blog posts, products, or regional content. Common mistakes we see: accidentally blocking CSS or JavaScript files (Google needs those to render pages properly), forgetting the trailing slash on directory rules, or using asterisks without understanding wildcard syntax. A 'Disallow: /*.pdf' blocks all PDFs, but plain 'Disallow: /file.pdf' only blocks that exact file. Test your rules in Google Search Console's robots.txt Tester before going live. At Ottawa SEO, we keep robots.txt minimal. Block only what genuinely shouldn't be indexed—admin panels, duplicate staging environments, thank-you pages, internal search result URLs. Overblocking is more dangerous than underblocking because you can accidentally hide important pages from Google. When in doubt, use meta robots noindex tags on individual pages instead of site-wide Disallow rules. The file is public, so never put sensitive paths in it; that's a roadmap for bad actors.