A crawl error occurs when a search engine bot attempts to access a page on your site but fails due to server issues, broken URLs, DNS problems, or access restrictions. These errors prevent indexing and signal potential site-health problems that directly impact organic visibility.
When Googlebot or another crawler visits your site, it sends HTTP requests to your server just like a browser. A crawl error happens when that request fails to return the expected content. The bot records the error type, timestamp, and affected URL, then decides whether to retry immediately, schedule another attempt, or mark the URL as inaccessible. Site-level errors prevent the bot from reaching your server entirely—DNS resolution failures mean the domain can't be found, server timeouts indicate your host isn't responding within the crawler's wait threshold, and robots.txt fetch failures block the bot from reading your crawl directives. URL-level errors occur after successful server connection but before content delivery: 404 status codes signal missing pages, 5xx server errors indicate backend problems, and redirect loops trap the bot in endless cycles. The crawler's behaviour differs by error type. Temporary server errors trigger retries over hours or days. Permanent 404s get marked as dead URLs after confirmation crawls. Robots.txt blocks stop crawling immediately and may lead to de-indexation of blocked pages over time.
Crawl errors directly impact which pages can rank. If a bot can't fetch a page, that page won't enter the index, no matter how well-optimized the content. For small sites with a few hundred pages, occasional errors have minimal impact—Google crawls frequently enough to catch corrections. For larger sites with tens of thousands of URLs, crawl errors consume crawl budget inefficiently. If the bot spends time repeatedly hitting broken URLs or timing out on slow pages, it has less capacity to discover new content or recrawl updated material. Infrastructure errors signal broader site-health issues. Persistent DNS failures might indicate registrar problems or misconfigured nameservers. Widespread 5xx errors suggest server capacity issues, database connection failures, or misconfigurations in your application stack. These problems affect real users too, not just bots. A sudden spike in crawl errors often precedes user-visible downtime. Monitoring crawl error patterns gives you early warning of infrastructure degradation before it cascades into lost traffic and conversions.
Google Search Console's crawl stats and indexing reports show error trends over time. The Coverage report categorizes URLs into valid, valid with warnings, excluded, and error states. Error states include submitted URLs not found (404s on sitemap URLs), server errors (5xx responses), redirect errors (chains or loops), and blocked resources. Each error entry shows the last crawl date and example URLs. Click through to see the specific HTTP status code, redirect path if applicable, and referring pages or sitemaps that linked to the broken URL. The distinction between server errors and not found errors matters. Server errors are temporary—Google retries these URLs automatically. Not found errors are treated as permanent unless the page returns with a 200 status on a subsequent crawl. Check the crawl stats section for host status trends. Unexpected drops in pages crawled per day often correlate with server timeout spikes or robots.txt fetch failures. You can also see crawl response codes over time: a sudden increase in 5xx responses points to backend instability, while a gradual rise in 404s might indicate link rot or a migration issue.
Certain patterns reveal specific problems. A 404 spike after a site migration usually means old URLs weren't redirected properly. Export the error list, compare it against your redirect map, and add missing 301 redirects to your server configuration. If crawl errors cluster around a specific subdirectory or URL pattern, check permissions and server rules affecting that section—an overly restrictive htaccess rule or application middleware might be blocking bots while allowing browser traffic. DNS errors affecting the entire domain typically stem from expired domains, incorrect nameserver settings, or registrar-level issues. Check your domain's WHOIS record and verify nameservers point to your hosting provider. Timeout errors concentrated during specific hours often indicate resource contention—your server can't handle peak traffic, or a background process consumes CPU during scheduled jobs. Soft 404s appear when your server returns a 200 status code but the page content signals non-existence—thin content, missing product pages, or search result pages with no results. Google detects these through content analysis and flags them as crawl errors even though the HTTP status is technically correct.
Not every crawl error demands immediate action. Legitimately deleted pages should return 404s—that's correct behaviour. If you discontinued a product or removed outdated content intentionally, the 404 is the right status code. Remove these URLs from your sitemap and let the errors clear naturally as Google recrawls. Blocked sections in robots.txt are expected. If you've disallowed admin panels, search filters, or duplicate parameter variations, those crawl errors are by design. Focus repairs on errors affecting valuable content. If a product page, service description, or blog post returns an error, fix it immediately—those URLs should be accessible and indexable. Prioritize errors on URLs receiving external backlinks or internal navigation links. A broken page linked from your main menu or referenced by high-authority external sites wastes link equity and damages user experience. For large error lists, segment by traffic potential. Use historical analytics to identify which broken URLs previously drove organic traffic, then prioritize their restoration or redirection. Errors on pages that never ranked and have no inbound links can often be deprioritized.
Regular technical audits catch errors before they accumulate. Run a full-site crawl with Screaming Frog or similar tools monthly, comparing results against previous crawls to spot new broken links or server errors. Implement automated uptime monitoring that checks not just homepage availability but also critical conversion paths and high-value content sections. Set up alerts for sudden changes in crawl error counts in Search Console—configure email notifications if server errors exceed a threshold or if newly submitted sitemap URLs return 404s. When making site changes, test thoroughly in staging. Before pushing a new URL structure, migration, or CMS update, crawl the staging environment and verify all redirects resolve correctly. Use version control and deployment checklists that include post-deployment crawl verification. Maintain a redirect map as a living document. Every time you remove or move a page, add the redirect to your map and implement it in your server configuration before deleting the old URL. For large sites, consider a redirect management system or database table that your application checks before serving 404s, automatically routing old URLs to their closest current equivalents.
A crawl error means the bot couldn't fetch the page at all—the HTTP request failed or returned an error status code. An indexing error occurs when the bot successfully crawled the page but chose not to index it, often due to noindex tags, canonical directives pointing elsewhere, or duplicate content filters. Crawl errors prevent the bot from seeing your content; indexing errors mean the bot saw it but decided not to include it in search results.
Retry timing depends on error type and site authority. For temporary server errors like 503 status codes, Google typically retries within hours. For 404 errors, the bot may recrawl after several days to confirm the page is truly gone. High-authority sites with frequent updates get more aggressive retry schedules. You can request immediate recrawling through Search Console's URL inspection tool if you've fixed a critical error and need faster reprocessing.
Google doesn't apply a direct penalty for crawl errors, but the impact is indirect and significant. If important pages can't be crawled, they disappear from the index and lose rankings. Persistent site-wide errors signal poor site quality, which can affect how Google allocates crawl budget and trusts your domain. User experience also suffers—if visitors encounter the same broken pages as bots, engagement metrics drop, which feeds into ranking algorithms over time.
No. Redirecting legitimately deleted pages to irrelevant destinations creates soft 404s and confuses users. Only redirect when there's a logical replacement—if you discontinued a product, redirect to the category page or a similar current product. If the content is truly gone with no equivalent, serve a proper 404 and remove the URL from sitemaps and internal links. Google expects some 404s on any healthy site; they're only problematic when they affect pages that should be accessible.
A sharp increase in 5xx server errors typically points to infrastructure problems: server resource exhaustion from traffic spikes or inefficient code, database connection failures, misconfigured deployments, or hosting provider issues. Check your server logs around the time the errors began, monitor CPU and memory usage, and review any recent code or configuration changes. If errors coincide with peak traffic hours, you may need to optimize server capacity or caching layers.
Cross-reference crawl errors with backlink profiles and historical traffic data. Use Search Console to export error URLs, then check which ones have external backlinks in Ahrefs or your preferred tool. Pull analytics data to see which broken URLs previously drove organic traffic. Prioritize fixing errors on pages linked from your main navigation, referenced in sitemaps, or tied to active marketing campaigns. Errors on orphaned pages with no links and no traffic history can usually wait or be left as legitimate 404s.