Screaming Frog is a desktop crawler that audits your site's technical SEO by mimicking how search engines discover and index pages. For agencies and in-house teams, it surfaces indexation barriers, redirect chains, broken assets, and metadata gaps faster than manual checks or server log parsing alone.
Search engines rely on crawlers to discover, parse, and index content. When your site presents broken links, orphaned pages, or metadata conflicts, you hand Google reasons to deprioritize or skip URLs entirely. Screaming Frog replicates this crawl behavior on your desktop, letting you see exactly what a bot encounters before it costs you rankings. Unlike browser-based checks, the tool follows every internal link, respects robots.txt and meta directives, and logs response codes, redirect hops, and resource load times in a single pass. For agencies running audits across dozens of domains or in-house teams inheriting legacy platforms, this centralized view replaces guesswork with actionable data. You identify canonical tag mismatches, discover pages accidentally blocked by noindex, and spot redirect chains that leak authority. The crawl becomes your source of truth for what exists, what's accessible, and what's misconfigured—long before traffic or rankings signal a problem.
The free version crawls up to 500 URLs per session, which suffices for small business sites, single-product landing pages, or exploratory spot-checks. Once you manage e-commerce catalogs, news archives, or multi-language directories, you hit that ceiling mid-crawl and lose visibility into deeper pages where indexation and duplicate content issues often hide. The paid license removes the URL limit, enables JavaScript rendering to crawl single-page applications accurately, and unlocks scheduled crawls that run overnight or weekly without manual intervention. You also gain custom extraction via XPath or regex, letting you pull schema markup, hreflang tags, or proprietary tracking parameters into columns for bulk validation. For agencies billing audit or retainer services, the license cost pays for itself in time saved and error reduction. If your workflow involves comparing staging to production, tracking redirect migrations, or generating recurring technical reports, the free tier becomes a false economy. Budget the license as infrastructure, not a luxury.
Screaming Frog offers spider mode, which follows links and respects directives like a traditional bot, and list mode, which crawls a predefined URL list without discovering new pages. Spider mode reveals your site's true link graph, exposing orphaned pages that lack any internal path and showing how deep users or bots must click to reach key content. List mode proves useful when auditing a specific subset—say, all product URLs from a sitemap or a batch of recently published articles. The rendering toggle determines whether the crawler executes JavaScript before extracting content. Many modern sites render titles, canonical tags, or navigation client-side, so crawling without JavaScript can return empty titles or miss links entirely. Enabling rendering slows the crawl but ensures you see what Google's WRS-based indexing pipeline actually processes. Compare a rendered and non-rendered crawl side by side to catch discrepancies, especially if your CMS injects metadata or schema via React or Vue. The tradeoff is speed versus accuracy; for audits that inform major decisions, accuracy wins.
After a crawl completes, the Overview tab summarizes response codes, page titles, meta descriptions, headings, and directives. Look at the status code breakdown: a healthy site shows most URLs returning 200, with intentional 301s for retired content and minimal 404s from recently removed pages. A spike in 302 temporary redirects often signals a migration mistake or CMS misconfiguration that leaks authority. Missing or duplicate title tags flag pages competing for the same query or pages Google may ignore in favor of better-optimized alternatives. Thin content warnings highlight pages below your word-count threshold, useful for spotting template bloat or auto-generated archives. Clicking any summary row filters the URL list, letting you export just the offending pages for a developer ticket. Prioritize fixes by impact: resolve indexation blockers like noindex tags on revenue pages first, then tackle redirect chains and missing alt text. The Overview tab turns a 10,000-row export into a ranked punch list, not a paralyzing spreadsheet.
The filter bar at the top of the URL list lets you isolate pages by folder, status code, content type, or custom criteria. If you manage a bilingual site, filter by /en/ and /fr/ to audit each language tree separately. For e-commerce, filter by /product/ to check canonical tags and structured data without noise from blog or legal pages. Combining filters—say, 200 status AND missing meta description—narrows results to actionable subsets. Once filtered, export to CSV or Google Sheets with only the columns you need: URL, title, word count, indexability. This focused export becomes a task list for your content team or a before-and-after comparison after remediation. Screaming Frog also supports custom search via regex, so you can find all URLs containing a tracking parameter, a legacy domain fragment, or a specific schema type. Bulk exports paired with pivot tables or VLOOKUP let you cross-reference crawl data against Analytics goal completions or Search Console impressions, surfacing high-traffic pages with technical debt or orphaned pages that once ranked but lost internal links during a redesign.
Screaming Frog connects to Google Analytics and Search Console APIs, appending sessions, goal completions, impressions, and clicks to each crawled URL. This integration reveals pages that receive organic traffic despite missing title tags, or URLs crawled by Google but never clicked, signaling a metadata or relevance problem. You can sort by sessions descending to prioritize technical fixes on high-traffic pages first, ensuring every hour of dev time protects revenue. Conversely, pages with zero sessions and zero clicks but strong internal linking might be orphaned by user behavior, not crawl depth—prompting a navigation or content strategy discussion. Export the merged dataset to calculate technical debt per traffic tier: how many top-100 landing pages have slow load times, duplicate H1s, or broken images? The API pulls are read-only and respect your Analytics view filters, so the data matches what stakeholders already monitor. For agencies running monthly retainers, this unified crawl plus performance snapshot becomes the foundation of a recurring health report, showing whether technical improvements correlate with ranking or traffic shifts without inventing a single metric.
New users often launch a crawl with default settings and drown in false positives. Crawling staging or dev subdomains without filtering inflates the URL count and surfaces issues that don't affect production. Ignoring the configuration tab means the tool may follow external links, crawl PDFs, or treat query parameters as unique pages, ballooning the dataset. Set a custom user-agent or respect crawl-delay if you're auditing a live site on shared hosting to avoid triggering rate limits. Forgetting to enable JavaScript rendering on a React or Next.js site returns incomplete data, leading you to flag missing content that actually exists post-render. On the flip side, rendering everything slows large crawls unnecessarily if your CMS outputs complete HTML server-side. Another pitfall: exporting every column and sorting by URL instead of by issue type, which buries critical problems under formatting noise. Treat the crawl configuration as a hypothesis—what specific technical question are you answering?—and tailor settings, filters, and exports to that question. A well-scoped 5,000-URL crawl with three targeted exports beats a shapeless 50,000-URL dump every time.
Crawl after major releases, CMS upgrades, or template changes to catch regressions immediately. For large or frequently updated sites, schedule a full crawl monthly and spot-check high-priority folders weekly. If you publish daily, a nightly crawl of new URLs ensures metadata and indexability settings deploy correctly before Google discovers them.
No. Screaming Frog shows what a bot could crawl if it follows your internal links, but logs reveal which pages Googlebot actually requested, how often, and which returned errors or timeouts. Use crawls to design your ideal link structure and logs to verify Google agrees. Together they diagnose crawl budget waste or orphaned pages ignored by real bots.
Set a realistic crawl speed to avoid server strain, enable JavaScript rendering if your CMS relies on client-side frameworks, and configure the user-agent to match Googlebot if you suspect cloaking or mobile-specific content. Exclude staging subdomains, ignore external links unless checking outbound quality, and limit crawl depth or URL count if you only need a sample.
Crawl the old site pre-launch and export URLs, titles, and meta descriptions. Crawl the new site and compare the exports to ensure every old URL has a matching 301, no pages lost indexability, and metadata carried over or improved. Filter for 404s or redirect chains introduced during the move. Re-crawl post-launch to confirm the live environment matches staging.
The free version caps at 500 URLs. Beyond that, check if robots.txt or noindex meta tags block deeper pages, if orphaned pages lack internal links, or if infinite pagination creates a crawl trap. Adjust the crawl settings to follow pagination parameters or increase max crawl depth. Review the log to see if the spider encountered server errors or timeouts.
Yes. The paid license includes command-line mode, letting you script crawls via cron or Task Scheduler and output results to a designated folder. Combine this with automated exports to Google Sheets, then layer on Apps Script or Python to flag new issues and email summaries. This turns Screaming Frog into a monitoring layer, not just a one-time audit tool.