Robots.txt ek plain text file hai jo aapke domain ke root level par hoti hai (example.com/robots.txt). Iska kaam hai search engine bots (Google, Bing, etc.) ko instructions dena ki website ke kaunse parts ko access karna allowed hai aur kaunse nahi. Ye crawl control ka sabse basic tool hai. Jab koi search engine bot aapki site par aata hai, sabse pehle wo robots.txt file check karta hai. Agar aapne kisi URL ko disallow kiya hai, to bot use crawl nahi karega. For example, "Disallow: /admin/" likhne se admin area crawl nahi hoga. Ye important hai kyunki aap nahi chahte ki login pages, duplicate staging areas, ya internal search results Google mein index ho. Common use cases include blocking parameter URLs that waste crawl budget, hiding development folders, preventing crawler overload on resource-heavy pages, and keeping private sections out of search. For a 500-page site, proper robots.txt can focus crawlers on high-value content instead of infinite filter combinations or session IDs. Key syntax: "User-agent: *" means all bots, "User-agent: Googlebot" targets only Google. "Disallow: /" blocks everything, "Disallow:" (blank) allows everything. You can also specify sitemap location with "Sitemap: https://example.com/sitemap.xml" to help crawlers discover your content structure faster. At Ottawa SEO Inc., we audit robots.txt files across our 500-domain portfolio because one wrong line can accidentally block entire sections from indexing. Common mistakes include blocking CSS or JavaScript files (hurts rendering), using robots.txt for security (it's publicly visible, anyone can read it), or blocking pages you actually want ranked. If you need a page out of search results but must allow crawling for internal links, use noindex meta tags instead of robots.txt disallow. Test changes at google.com/webmasters/tools/robots-testing-tool before deploying live. A well-configured robots.txt typically saves 15-30% of wasted crawl budget on mid-sized sites.