A robots.txt file sits at yoursite.com/robots.txt and acts as the first checkpoint for search engine bots. When Googlebot or any crawler arrives, it checks this file before accessing anything else. You use it to block crawlers from wasting time on admin pages, duplicate content, or sections you don't want indexed. The syntax is simple. A "User-agent" line specifies which bot the rule applies to (or * for all), and "Disallow" lines list paths to block. For example, "Disallow: /admin/" keeps bots out of your admin panel. You can also use "Allow" to permit specific subdirectories within a blocked path, and "Sitemap" to point crawlers to your XML sitemap location. The biggest mistake we see is using robots.txt to hide sensitive content. It blocks crawling but doesn't prevent indexing if someone links to the page. For actual confidentiality, use password protection or noindex meta tags. Another common error is accidentally blocking critical resources like CSS or JavaScript files that Google needs to render pages properly, which tanks your mobile-friendliness scores. Crawl budget matters most for larger sites. If you're running 500+ pages, blocking tag archives, search result pages, or printer-friendly versions keeps Google focused on your money pages. On a 50-page small business site, robots.txt is less critical but still useful for blocking thank-you pages or internal search. At Ottawa SEO, we audit robots.txt files during every technical review. We've seen clients lose 40% of their indexed pages overnight from a single wildcard typo. We also check that the file isn't accidentally blocking the sitemap or disallowing trailing slashes when the CMS uses them. Test changes in Google Search Console's robots.txt tester before pushing live, and remember that changes aren't instant—crawlers cache the file for up to 24 hours. Keep it lean, specific, and always verify it's doing what you think it's doing.