Screaming Frog SEO Spider: What It Shows and What It Misses

Screaming Frog SEO Spider is a website crawler that maps your site’s technical architecture, surfacing broken links, duplicate content, missing metadata, redirect chains, and a range of other on-page signals that affect how search engines process your pages. Most SEO professionals use it as a first-pass diagnostic, a way to get a structured picture of a site before deciding where to spend time.

It is one of the most practically useful tools in technical SEO. It is also one of the most frequently misread. What Screaming Frog gives you is a crawl-level inventory of your site’s condition. What it does not give you is an interpretation of what that condition means for your rankings, your traffic, or your revenue. That gap matters more than most people acknowledge.

What Screaming Frog Actually Does

Screaming Frog works by simulating how a search engine bot moves through your site. It follows links from a starting URL, catalogues every page it finds, and records a set of attributes for each one: status codes, title tags, meta descriptions, heading structure, word count, canonical tags, hreflang attributes, response times, and more. The free version crawls up to 500 URLs. The paid licence removes that cap and adds integrations with Google Analytics, Google Search Console, PageSpeed Insights, and a handful of other data sources.

The output is a spreadsheet-style interface where you can filter, sort, and export data across dozens of dimensions. You can view all pages returning a 404 status. You can find every page with a title tag over 60 characters. You can identify redirect chains where a URL passes through three or more hops before reaching its destination. You can see which pages are blocked from crawling via robots.txt and which ones are canonicalised away from themselves.

None of this is glamorous. It is infrastructure work, the equivalent of walking through a building with a clipboard before deciding what needs repair. I have run this audit process on sites ranging from a 200-page professional services firm to an e-commerce catalogue with north of 400,000 URLs. The volume changes the complexity of the crawl, but the fundamental logic stays the same: you are building a map before you start giving directions.

If you want a broader framework for where technical audits fit within a complete SEO programme, the SEO strategy hub covers the full picture from positioning to measurement.

The Issues Screaming Frog Surfaces Most Reliably

There are a handful of technical problems that Screaming Frog finds consistently and accurately, and these are worth understanding in some depth because they represent the genuine value of the tool.

Broken internal links. Any link on your site pointing to a URL that returns a 4xx or 5xx status is flagged. Broken internal links waste crawl budget, create poor user experiences, and signal to search engines that a site is poorly maintained. On large sites, these accumulate quietly over time as pages are deleted or restructured without updating the links pointing to them. I have audited sites where the team was genuinely surprised to find several hundred broken internal links. Nobody had deleted them maliciously. They had just accumulated across years of content updates with no systematic review.

Redirect chains and loops. A redirect chain occurs when URL A redirects to URL B, which redirects to URL C. Each hop in that chain adds latency and dilutes link equity. Screaming Frog traces these chains and shows you the full path, which makes it straightforward to identify where redirects can be consolidated into a single hop. Redirect loops, where two URLs redirect to each other, are rarer but cause crawlers to abandon the path entirely.

Duplicate content and canonicalisation issues. Screaming Frog identifies pages with identical or near-identical title tags, and it surfaces canonicalisation conflicts where a page’s canonical tag points somewhere unexpected. This is particularly relevant on e-commerce sites where product pages with minor variations can generate dozens of near-duplicate URLs if faceted navigation is not managed carefully.

Missing and malformed metadata. Every page without a title tag, every page with a meta description that exceeds 155 characters, every page missing an H1, these are flagged in the overview. This is useful as a quality-control mechanism, particularly after a site migration or a CMS change where template errors can silently strip metadata from large numbers of pages.

Crawlability and indexability signals. Screaming Frog distinguishes between pages that are blocked by robots.txt, pages with noindex directives, and pages that are canonicalised to another URL. This distinction matters enormously. A page blocked by robots.txt cannot be crawled. A page with noindex can be crawled but will not appear in search results. Confusing these two states is a common mistake, and it leads to incorrect diagnoses about why certain pages are not ranking.

Where the Tool’s Limitations Start

Screaming Frog is a crawler. It sees what a bot sees. It does not see what Google actually does with that information, and that gap is wider than the SEO industry often admits.

A crawl report will tell you that a page has a thin word count. It will not tell you whether Google considers that page thin relative to the query it is targeting. A 300-word page that directly answers a specific question may outperform a 2,000-word page that buries the answer in padding. Word count is a proxy metric. Screaming Frog surfaces the proxy. The interpretation requires judgment.

Similarly, the tool will flag every instance of a missing meta description. In practice, Google rewrites meta descriptions for a significant proportion of queries regardless of what you have written. Fixing every missing meta description on a 50,000-page site is a substantial piece of work. Whether it moves any meaningful metric is a different question, and one that Screaming Frog cannot answer for you.

I spent time judging the Effie Awards, reviewing campaigns that had generated measurable commercial results. The ones that stood out were not the ones with the cleanest technical setups. They were the ones where the strategy was right and the execution was coherent. Technical SEO is a prerequisite for performance, not a driver of it. You need the foundations to be sound, but sound foundations do not guarantee you will build anything worth visiting.

This connects to a broader point about how SEO tools get used in agency environments. When I was running agency teams, I noticed that junior analysts would often conflate the volume of issues found with the quality of the audit. A 47-page crawl report with 300 flagged issues looks thorough. It may also be a list of things that will collectively move your organic traffic by approximately nothing. The discipline is in knowing which issues matter, and that requires connecting crawl data to performance data.

How to Connect Crawl Data to Performance Data

Screaming Frog’s integrations with Google Search Console and Google Analytics are where the tool becomes genuinely useful for commercial decision-making, rather than just technical inventory.

When you connect Google Search Console, each URL in the crawl is enriched with impressions, clicks, average position, and click-through rate data. This means you can filter your crawl results to show only pages that are currently receiving search impressions, and then look at the technical issues present on those specific pages. A broken internal link on a page with zero impressions is low priority. A broken internal link on a page receiving 10,000 impressions per month is a different conversation.

The Google Analytics integration adds session data and conversion data to the same view. Now you can see not just which pages have technical issues, but which of those pages are driving revenue or leads. This is the layer of analysis that turns a crawl report into a prioritised action plan.

The practical workflow I have used across multiple site audits runs as follows. First, crawl the site and export the full URL list with all available attributes. Second, connect Search Console and Analytics data to the crawl. Third, filter the URL list to pages with meaningful traffic or conversion contribution. Fourth, review the technical issues present on those pages specifically. Fifth, build a prioritised fix list based on the combination of issue severity and page-level commercial value. Anything outside that filtered set gets a separate, lower-priority review.

This sounds straightforward. In practice, most teams skip steps two and three and go straight to generating a fix list from the raw crawl data. That is how you end up spending three sprints fixing meta descriptions on pages that nobody visits.

Using Screaming Frog for Site Migrations

Site migrations are where Screaming Frog earns its place most clearly. A migration without a proper crawl audit before and after is a risk that most businesses do not fully appreciate until they have lost 40% of their organic traffic in the first month post-launch.

The pre-migration crawl establishes a baseline. You export every URL on the current site, every redirect in place, every canonical relationship, and every metadata value. This becomes your reference document. When the new site goes live, you crawl it again and compare the two exports. Any URL that existed in the pre-migration crawl and does not have a corresponding redirect or equivalent page in the new site is a gap. Any redirect that has been set up incorrectly, pointing to a 404 rather than a live page, is a risk.

I have been involved in migrations where the redirect mapping was done carefully and the post-launch crawl still surfaced 200 URLs that had been missed. On a site with tens of thousands of pages, that is not negligence. It is the normal outcome of complex work done under time pressure. The crawl catches what human review misses.

Screaming Frog also has a staging crawl feature that lets you crawl a site before it goes live by adjusting the user agent or crawling via a staging environment. This allows you to verify redirect implementation and metadata before launch rather than after, which is the only sensible approach.

Visualising Site Architecture

One feature that gets less attention than it deserves is Screaming Frog’s site architecture visualisation. The tool can generate a crawl map showing the link depth of every page on your site, meaning how many clicks it takes to reach each URL from the homepage.

Link depth matters for two reasons. First, pages buried deep in the site structure receive fewer internal links and are therefore harder for crawlers to discover and prioritise. Second, pages at greater depth tend to receive less PageRank distribution from the homepage, which affects their ability to rank for competitive queries.

On large e-commerce sites, it is common to find important category pages sitting at four or five clicks from the homepage because the navigation structure was built around user experience principles without considering crawlability. The visualisation makes this visible in a way that a spreadsheet does not. When you can see that your highest-revenue category is three levels deeper than a set of thin blog posts, the structural problem becomes hard to ignore.

This kind of architectural analysis connects directly to internal linking strategy. Screaming Frog shows you which pages have the most internal links pointing to them. Comparing that list against your commercially important pages often reveals a mismatch. The pages that matter most to the business are not always the pages that the site’s internal link structure treats as most important.

Custom Extraction and Configuration

Beyond the standard crawl attributes, Screaming Frog allows custom extraction using XPath, CSS selectors, or regex. This means you can pull specific elements from pages that the default crawl does not capture.

Practical applications include extracting structured data markup to verify that schema is implemented correctly across all pages, pulling specific on-page elements like price or availability data from product pages, or extracting custom attributes relevant to a specific CMS implementation. For sites with complex template structures, custom extraction allows you to verify that dynamic content is rendering correctly in the crawler’s view.

The JavaScript rendering mode is worth mentioning here. By default, Screaming Frog crawls pages without executing JavaScript, which is how older search engine bots behaved. Modern Googlebot does render JavaScript, but not always immediately and not always completely. The JavaScript crawl mode in Screaming Frog uses a headless browser to render pages as a modern bot would, allowing you to identify content or links that only appear after JavaScript execution. For sites built on React, Angular, or other JavaScript frameworks, this is not optional. It is the only way to get an accurate picture of what the crawler actually sees.

The skill set required to get full value from these advanced features is not trivial. Moz has written about the soft skills that separate effective SEO practitioners from those who produce technically competent but commercially irrelevant work. The ability to configure a crawler and interpret its output is a hard skill. Knowing which output to act on is something else entirely.

Screaming Frog in the Context of a Broader SEO Audit

Screaming Frog is one tool in an audit stack, not a complete audit process. A thorough technical SEO review will also draw on Google Search Console for index coverage and Core Web Vitals data, a backlink analysis tool for external link profile assessment, a keyword rank tracker for positioning data, and log file analysis for understanding how Googlebot is actually crawling the site as opposed to how it theoretically should be.

Log file analysis is particularly underused. Screaming Frog has a companion tool, Screaming Frog Log File Analyser, which processes server log files and shows you exactly which URLs Googlebot requested, how frequently, and what status codes it received. This data often contradicts what you would expect from the crawl alone. Pages you assumed were being crawled regularly may show minimal bot activity. Pages you assumed were low priority may be receiving disproportionate crawl attention.

The combination of crawl data and log data gives you a more complete picture of how search engines are actually engaging with your site, as opposed to how they should be engaging with it based on your architecture. That distinction matters when you are trying to diagnose why a technically sound site is underperforming in search.

There is a version of SEO that treats tools as the answer rather than as instruments for finding answers. I have seen it in agencies where the deliverable was a Screaming Frog report with a long list of issues, presented as evidence of thoroughness. The client received a document. The client’s rankings did not change. Explaining the commercial value of SEO work requires connecting technical findings to business outcomes, and that connection is not visible in a crawl report alone.

The broader point about SEO as a business function, rather than a technical exercise, runs through everything on the Complete SEO Strategy hub. Technical audits matter. They matter because they affect how search engines process and rank your content. They do not matter in isolation from the strategic questions about what you are trying to rank for, who you are trying to reach, and what you want them to do when they arrive.

A Practical Starting Point for New Users

If you are using Screaming Frog for the first time, the temptation is to crawl your site, look at the overview tab, and start working through every flagged issue in sequence. That approach will keep you busy. It will not necessarily improve your organic performance.

A more useful starting point is to crawl your site and immediately filter the results to show only pages that are indexable, meaning pages without noindex directives and without canonical tags pointing elsewhere. These are the pages that can actually appear in search results. Everything else is secondary.

Within that indexable set, sort by inlinks, meaning the number of internal links pointing to each page. Pages with few internal links pointing to them are structurally deprioritised by your own site. If those pages are commercially important, that is a structural problem worth addressing before you spend time on metadata formatting.

Then connect your Search Console data and look at the pages in that indexable set that are generating impressions but low clicks. These are pages where your metadata is potentially underperforming relative to the search demand that exists. Title tag and meta description work on these specific pages is likely to have a measurable effect. The same work on pages with zero impressions will have no measurable effect at all.

This is not a comprehensive audit methodology. It is a way of getting to the work that matters without first completing the work that does not. Most sites have more issues than they have capacity to address. The discipline is in the triage, not the volume of findings. Tools like Screaming Frog are genuinely useful. How you use them is the variable that determines whether they produce results or just reports.

About the Author

Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.

Frequently Asked Questions