The robots.txt file sits in your WordPress root directory, the same folder that contains wp-config.php, wp-content, and wp-admin. You access it at yourdomain.com/robots.txt in any browser. WordPress creates a virtual robots.txt automatically if you haven't uploaded a physical file. This virtual version blocks only wp-admin by default and won't show up in your file manager or FTP client because it's generated on the fly by WordPress core. To create or edit a physical robots.txt file, connect via FTP or use your hosting control panel's file manager. Navigate to public_html or the equivalent root directory for your domain. Create a plain text file named robots.txt and upload it. Once a physical file exists, WordPress stops generating the virtual version and serves your file instead. Most sites need a custom robots.txt to block crawlers from wp-json, feed URLs, search result pages, and other WordPress-specific paths that waste crawl budget. A typical setup looks like this: - User-agent: * - Disallow: /wp-admin/ - Disallow: /wp-json/ - Disallow: /?s= - Disallow: /feed/ - Sitemap: https://yourdomain.com/sitemap.xml Plugins like Yoast SEO and Rank Math let you edit robots.txt from the WordPress dashboard under their tools section. This is easier than FTP but still writes to the same root directory file. The plugin approach works fine for most sites, but we prefer direct file access at Ottawa SEO when managing 500+ domains because it's faster to audit and doesn't depend on a plugin staying active. One gotcha: if your site is set to discourage search engines under Settings > Reading, WordPress adds Disallow: / to the virtual robots.txt, blocking everything. Check that setting first if Google suddenly stops indexing your pages. Also verify your hosting setup actually serves the root directory at your primary domain, because some multisite or subdirectory installs can create confusion about which robots.txt file applies.