A text file at site root that tells crawlers which URLs they can and cannot access.
If you're researching What is Robots.txt, this guide gives you the practitioner-grade reference our own team uses internally. **robots.txt** — A text file at site root that tells crawlers which URLs they can and cannot access.
robots.txt blocks crawling, not indexation. Pages blocked in robots.txt can still appear in search results if linked externally. Use noindex meta tag to truly remove a page from results. Practical tip: most teams encounter this concept when troubleshooting indexing or ranking issues — knowing the canonical definition saves hours of misdiagnosis. This term appears frequently in modern SEO documentation and in the Search Console help center; understanding it well prevents common configuration mistakes that cost rankings.
Robots.txt sits in the **Technical SEO** layer of search engine optimization. Understanding it correctly is essential for anyone working on technical SEO, content strategy, or executing campaigns at the level required to compete in modern search results.
The single most common mistake practitioners make with robots.txt is treating it as a tactic in isolation, rather than as one signal among hundreds that Google evaluates. Done well, robots.txt contributes to compound ranking gains; done poorly, it creates technical debt that handicaps every future SEO investment. FAQ on "what is robots.txt" — the short version is below the technical primer. This term appears frequently in modern SEO documentation and in the Search Console help center; understanding it well prevents common configuration mistakes that cost rankings.
When implementing robots.txt, the highest-leverage practices are:
- Treat robots.txt as a foundation, not a bolt-on. Get it right at the architectural level rather than retrofitting later. - Audit existing implementations regularly — Google's interpretation of robots.txt evolves with each algorithm update. - Validate technical implementations using Google's official tools (Search Console, Rich Results Test, PageSpeed Insights) before assuming success. - Document your approach so future site changes don't accidentally break robots.txt configuration. - Measure outcomes against actual ranking and traffic data, not vanity metrics. FAQ on "what is robots.txt" — the short version is below the technical primer. If you're implementing this concept on your own site, the documentation linked at the bottom of this page covers the technical specifics in greater depth.
The most frequent errors we see clients make with robots.txt:
1. **Treating it as a checkbox item.** robots.txt is rarely a one-time setup — it requires ongoing maintenance as content, code, and Google's standards evolve. 2. **Implementing without measurement.** Without tracking the impact of robots.txt changes, you can't distinguish what's working from what's noise. 3. **Following outdated advice.** SEO tactics around robots.txt have changed substantially over the years — guides published before 2023 frequently recommend approaches that are now ineffective or actively harmful. 4. **Over-optimizing.** Excessive focus on a single signal almost always backfires. robots.txt works in concert with other ranking factors. FAQ on "what is robots.txt" — the short version is below the technical primer.
These terms are closely related to robots.txt and worth understanding in context:
- **Crawling** — The process by which search engine bots discover web pages by following links. - **Noindex Tag** — A meta tag (or X-Robots-Tag header) telling search engines not to index a page. If you're implementing this concept on your own site, the documentation linked at the bottom of this page covers the technical specifics in greater depth. This term appears frequently in modern SEO documentation and in the Search Console help center; understanding it well prevents common configuration mistakes that cost rankings.
If you're trying to improve your site's performance with respect to robots.txt, the most useful next step is a no-pressure technical audit. We'll examine your current implementation, identify gaps, and walk through the specific improvements that would deliver the highest ROI for your business.
Book a free strategy call or read our broader SEO methodology to see how we approach work like this for technical seo clients across Canada and the US. Practical tip: most teams encounter this concept when troubleshooting indexing or ranking issues — knowing the canonical definition saves hours of misdiagnosis. If you're implementing this concept on your own site, the documentation linked at the bottom of this page covers the technical specifics in greater depth.
Yes — robots.txt is part of the Technical SEO layer of search engine optimization, and it influences how search engines crawl, index, and rank your pages.
Implementation depends on your tech stack and CMS. For most sites, robots.txt is best handled at the template level so it applies consistently across new content.
Google's official documentation is the authoritative source. We've also covered robots.txt in our broader SEO content — see related terms below.