Robots.txt works as a set of instructions for search engine crawlers, sitting at yoursite.com/robots.txt. When a bot visits your site, it checks this file first to see what it's allowed to crawl. You write simple directives like "User-agent: *" to target all bots or "User-agent: Googlebot" for specific ones, then add "Disallow: /admin/" to block access to certain paths. The file uses pattern matching. "Disallow: /private/" blocks everything under that folder. "Disallow: /*.pdf$" blocks all PDFs. "Allow: /public/" can override a broader disallow rule for subfolders. Most crawlers respect these rules, but robots.txt is not a security mechanism—malicious bots ignore it, and the file itself is publicly readable. Common mistake: blocking a page in robots.txt doesn't remove it from Google's index. If the URL is already indexed or has inbound links, Google may still list it with a snippet like "A description is not available because of this site's robots.txt." To actually deindex content, you need a noindex meta tag or X-Robots-Tag header, and the page must be crawlable so Google can see that tag. At Ottawa SEO, we see sites accidentally blocking their own CSS and JavaScript files in robots.txt, which breaks Google's rendering and mobile-first indexing. We also see overly aggressive disallows that block faceted navigation or pagination—sometimes you want to block indexing but allow crawling so link equity flows. The syntax is unforgiving: one typo in a wildcard pattern can expose admin areas or waste crawl budget on infinite calendar pages. Key directives include "Crawl-delay" for non-Google bots and "Sitemap: https://yoursite.com/sitemap.xml" to point crawlers to your XML sitemap. Test your robots.txt in Google Search Console's tester tool before deploying, especially after site migrations or CMS changes. A misconfigured robots.txt can deindex your entire site in hours. How Search Engines Read Robots.txt During Crawling When Googlebot or Bingbot arrives at your domain, the first HTTP request typically goes to /robots.txt before any other URL gets fetched. The crawler caches this file—Google refreshes its cache roughly once per day, though this varies based on crawl frequency for your site. If your server returns a 5xx error for robots.txt, most crawlers pause crawling temporarily to avoid accessing pages they might not be allowed to reach. A 404 response means no restrictions exist, so everything becomes crawlable. This fetch-first behavior is why a broken robots.txt can halt indexing entirely: the crawler waits for valid instructions rather than assuming permission. Understanding this sequence matters when debugging sudden drops in crawl activity after server configuration changes. Robots Txt Directives Explained for Common Use Cases Beyond basic Disallow rules, directives follow a specific precedence. More specific path rules override broader ones, so Allow: /blog/public-post can grant access even when Disallow: /blog/ exists higher in the file. The $ character anchors patterns to the end of URLs—useful for blocking file types without accidentally catching subfolders. User-agent declarations apply to all directives below them until the next User-agent line appears, creating logical groupings. Common directives include: - Disallow: / blocks the entire site for that user-agent - Allow: /api/public/ whitelists specific paths within blocked directories - Sitemap: declares sitemap location, though Google also accepts sitemaps via Search Console - Crawl-delay: respected by Bing and Yandex but ignored by Google Order matters within user-agent blocks, and whitespace errors can invalidate entire sections. How to Configure Robots.txt for Different CMS Platforms WordPress generates a virtual robots.txt by default, but plugins like Yoast or Rank Math let you customize it through the admin interface without FTP access. Shopify restricts robots.txt editing significantly—you can only modify it through theme liquid files, and certain paths remain blocked regardless of your preferences. Squarespace and Wix offer limited control through their SEO panels, often preventing granular directive customization. For sites on custom CMS platforms or static hosting, you upload the file directly to the web root. Always verify the file serves correctly with a browser request to yourdomain.com/robots.txt after deployment. Platform migrations frequently break robots.txt because the new system either overwrites your custom rules or serves the file from a different location than the old setup. Robots.txt Interaction with Crawl Budget and Site Architecture For sites under roughly 10,000 pages, crawl budget rarely becomes a practical concern—Googlebot typically handles smaller sites without issue. Larger sites benefit from strategic disallows that prevent crawlers from wasting requests on low-value URLs like internal search results, session-based parameters, or infinite scroll endpoints. However, blocking pages in robots.txt means link equity from those pages cannot flow outward, which differs from using noindex where the page remains crawlable. The tradeoff requires evaluating whether blocked sections contain valuable internal links worth preserving. Sites with complex faceted navigation often implement a hybrid approach: robots.txt blocks the most problematic parameter combinations while canonical tags and noindex handle cleaner variations that still need crawling for link purposes. Frequently Asked Questions How does robots.txt file work? The robots.txt file sits at your domain's root directory and provides crawling instructions to search engine bots. Crawlers request this file first, cache the directives, then follow the allow and disallow rules when deciding which URLs to fetch during their crawl session. How search engines read robots.txt? Search engines make an HTTP request to /robots.txt before crawling other pages. They cache the response for roughly a day, parse the user-agent declarations to find applicable rules, then apply matching allow and disallow patterns to every URL they consider fetching. Robots txt directives explained? Key directives include User-agent to specify which crawler the rules apply to, Disallow to block paths, Allow to override broader blocks for specific URLs, Sitemap to declare your sitemap location, and Crawl-delay to throttle request frequency for bots that honor it. How to configure robots.txt? Create a plain text file named robots.txt, add User-agent and Disallow lines following standard syntax, then upload it to your site's root directory. Test with Google Search Console's robots.txt tester before deploying, and verify it serves correctly via browser request.