What is a robots.txt file and what should I put in it?

What is a robots.txt file and what should I put in it?A robots.txt file tells search engine crawlers which parts of your site they may or may not crawl. Most sites need a minimal robots.txt that allows everything important and disallows admin/internal areas, plus a sitemap reference at the bottom.AnswerA robots.txt file is a plain-text file at the root of your domain (yoursite.com/robots.txt) that gives crawl instructions to search engine bots. It uses User-agent and Disallow/Allow directives following the Robots Exclusion Protocol. A typical sound robots.txt for a Canadian SMB site looks like: `User-agent: *` followed by Disallow rules for admin paths (`/wp-admin/`, `/admin/`, `/login/`), internal search results (`/?s=`), tracking-parameter URLs (`/*?utm_`), and a Sitemap directive at the bottom (`Sitemap: https://yoursite.com/sitemap.xml`). Common robots.txt mistakes: (1) Disallowing critical CSS or JavaScript directories — Google needs to render the page to evaluate it; (2) Disallowing the entire site during migration and forgetting to remove it (this happens regularly and is catastrophic); (3) Trying to use robots.txt to keep pages out of search — robots.txt prevents crawling, not indexing; pages can still appear in search if linked from elsewhere. Use the noindex meta tag instead. (4) Conflicting rules for different user agents that accidentally block Googlebot. Test changes with the robots.txt tester before deploying.