Screaming Frog Tutorial: How to Run a Complete Site Audit

Prerequisites

Before you start, make sure you have the following in place:

• Screaming Frog SEO Spider installed. The free version crawls up to 500 URLs. For larger sites or advanced features (crawl scheduling, JavaScript rendering, API integrations), you need the paid license at £259/year (approximately $329 USD as of 2025).
• Google Search Console access for the site you're auditing. You'll connect this to Screaming Frog to pull real crawl and performance data.
• Google Analytics 4 access (optional but recommended). Connecting GA4 lets you overlay traffic data on crawl results, so you can prioritize fixes by actual impact.
• A stable internet connection. Crawling a large site generates significant network traffic. Avoid running audits on metered connections.
• Sufficient RAM. Screaming Frog stores crawl data in memory by default. For sites over 10,000 URLs, allocate at least 4GB of RAM to the application. For sites over 100,000 URLs, switch to database storage mode (Configuration > System > Storage Mode).

Try Screaming Frog →

Step 1: Configure Crawl Settings

Don't just paste your URL and hit Start. Default settings work for quick checks, but a proper audit requires configuration.

Set the User Agent

Go to Configuration > User-Agent. For most audits, use Googlebot to simulate how Google sees your site. If you're auditing for Bing or another search engine, switch accordingly. This matters because some sites serve different content or apply different rules based on the user agent.

Adjust Speed Settings

Navigate to Configuration > Speed. The default setting (5 concurrent threads) is reasonable for most servers. If you're auditing a site on shared hosting or a server with limited resources, reduce to 1-2 threads and increase the delay between requests to 500ms-1000ms. Crawling too aggressively can slow down or crash a production server.

Configure Rendering

For sites that rely heavily on JavaScript (React, Angular, Vue, Next.js), go to Configuration > Spider > Rendering and switch from "Text Only" to JavaScript. This uses a built-in Chromium browser to render pages, which is slower but catches content and links that only appear after JavaScript execution. For static HTML sites, text-only crawling is faster and sufficient.

Set URL Scope

Under Configuration > Include/Exclude, add any patterns you want to skip. Common exclusions:

• /wp-admin/ and /wp-login.php (WordPress admin pages)
• /cart/ and /checkout/ (ecommerce transaction pages)
• /tag/ or /author/ (thin taxonomy pages you may not want to audit)
• Query parameter variations like ?sort= or ?filter=

This keeps the crawl focused on pages that matter for SEO.

Step 2: Connect API Integrations

This is where Screaming Frog becomes significantly more powerful than a basic crawler.

Google Search Console

Go to Configuration > API Access > Google Search Console. Authenticate with your Google account and select the property. This pulls in:

• Search impressions, clicks, CTR, and average position for each URL
• Index coverage data showing which pages Google has indexed
• Crawl data from Google's own infrastructure

This data transforms your audit from a list of technical issues into a prioritized action plan based on real search performance.

Try Google Search Console →

Google Analytics 4

Under Configuration > API Access > Google Analytics 4, connect your GA4 property. This adds sessions, engaged sessions, conversions, and other behavioral metrics to each crawled URL. Pages with high traffic but technical issues should be fixed first.

Try Google Analytics →

PageSpeed Insights

Enable Configuration > API Access > PageSpeed Insights with your Google API key. This runs Lighthouse audits on each URL during the crawl, giving you Core Web Vitals data (LCP, INP, CLS) alongside your technical findings. Note that this dramatically increases crawl time — consider using a sample for large sites.

Step 3: Run the Crawl

Enter your site's URL in the address bar at the top of Screaming Frog and click Start. For a site with 5,000 pages on a standard server, expect the crawl to take 15-45 minutes depending on your speed settings and whether JavaScript rendering is enabled.

While the crawl runs, monitor the bottom-right status bar. It shows:

• URLs crawled vs. URLs remaining in the queue
• Response time — if this spikes, your server may be struggling
• Memory usage — if this approaches your allocation limit, pause and increase it

Don't interact heavily with the application during the crawl. Let it finish completely before analyzing results.

Step 4: Analyze On-Page Elements

Once the crawl completes, start with the tabs across the main interface.

Page Titles

Click the Page Titles tab. Use the filter dropdown to check for:

• Missing — Pages with no title tag. These need one immediately.
• Duplicate — Multiple pages sharing the same title. Each page should have a unique title.
• Over 60 characters — Titles that will be truncated in search results. Aim for 50-60 characters.
• Below 30 characters — Titles that are too short and may not be descriptive enough.

Meta Descriptions

Switch to the Meta Description tab. Apply the same filters:

• Missing — Google will auto-generate these, but you'll get better CTR with custom descriptions.
• Duplicate — Each page should have a unique meta description.
• Over 155 characters — Will be truncated on desktop SERPs.

H1 Tags

Check the H1 tab for missing, duplicate, or multiple H1 tags per page. Every indexable page should have exactly one H1 that clearly describes the page content.

Step 5: Identify Crawl and Indexability Issues

Response Codes

Click the Response Codes tab. Filter by category:

• 3XX (Redirects) — Check for redirect chains (A → B → C). Each chain adds latency and dilutes link equity. Fix chains so they point directly to the final destination.
• 4XX (Client Errors) — Broken pages. Internal links pointing to 404s waste crawl budget and create dead ends for users and search engines.
• 5XX (Server Errors) — Server-side failures. These need immediate investigation, especially if they affect critical pages.

Directives

Under the Directives tab, check for pages that are unintentionally blocked:

• Noindex — Pages marked noindex won't appear in search results. Verify these are intentional.
• Canonicalized — Pages pointing their canonical tag to a different URL. Check that canonical targets are correct.
• Blocked by robots.txt — Pages that Googlebot can't access. Cross-reference with your robots.txt rules.

Step 6: Audit Internal Linking

Crawl Depth

Go to the Crawl Depth column in the Internal tab (or use the Crawl Depth report under Reports > Crawl Path). Your most important pages should be within 3 clicks of the homepage. Pages buried 5+ clicks deep are harder for search engines to discover and tend to receive less authority.

Orphan Pages

If you connected Google Search Console or submitted a sitemap, check for orphan pages — URLs that appear in your sitemap or Google's index but have no internal links pointing to them. These are effectively invisible to crawlers navigating your site structure.

In Screaming Frog 21+, find these under Reports > Orphan Pages.

Internal Link Distribution

Export internal link data via Bulk Export > Links > All Inlinks. Look for:

• Key pages with few internal links — Your most important pages should have the most internal links.
• Low-value pages with excessive links — Tag pages, archive pages, or thin content pages that are receiving disproportionate link equity.

Step 7: Check for Duplicate Content

Go to the URL tab and look at the Hash and Near Duplicates columns (enable near duplicate detection under Configuration > Content > Duplicates).

Common duplicate content patterns:

• HTTP and HTTPS versions of the same page
• www and non-www versions
• Trailing slash and non-trailing slash variations
• URL parameter variations (/page vs. /page?ref=twitter)
• Paginated pages duplicating content from page 1

For each set of duplicates, verify that proper canonical tags are in place pointing to the preferred version.

Step 8: Audit Images and Media

Click the Images tab. Filter for:

• Missing Alt Text — Images without alt attributes hurt accessibility and miss an opportunity for image search visibility.
• Over 100KB — Large images slow page load times. Look for images that should be compressed or converted to modern formats like WebP or AVIF.
• Missing Alt Attributes vs. Alt Text Over 100 Characters — Alt text should be descriptive but concise.

For a detailed image optimization workflow, export the full image list and cross-reference with PageSpeed Insights data.

Step 9: Review Structured Data

If your site uses schema markup, go to the Structured Data tab. Screaming Frog validates JSON-LD, Microdata, and RDFa against Schema.org specifications. Look for:

• Validation errors — Missing required fields or incorrect data types.
• Pages missing structured data — Product pages without Product schema, articles without Article schema, etc.
• Inconsistent implementation — Some pages have schema while similar pages don't.

Step 10: Export and Prioritize Findings

Generate the Crawl Overview Report

Go to Reports > Crawl Overview to generate a summary of all issues found. This report categorizes findings by type and severity, giving you a high-level view of site health.

Export Specific Issues

For each problem area, use Bulk Export to create spreadsheets you can share with developers or clients. Key exports:

• Response Codes > Client Error (4XX) Inlinks — Shows which internal pages link to broken URLs
• Directives > Noindex Inlinks — Pages linking to noindexed content
• Links > All Redirects — Complete redirect map

Prioritization Framework

Not all issues are equal. Prioritize fixes in this order:

1. Server errors (5XX) on high-traffic pages — immediate fix
2. Broken internal links to important pages — this week
3. Missing or duplicate title tags on indexable pages — this week
4. Redirect chains on high-authority pages — this sprint
5. Missing alt text, meta descriptions — ongoing maintenance
6. Minor optimization opportunities — backlog

If you connected GA4 and Search Console, sort issues by traffic impact. A missing title tag on a page with 10,000 monthly sessions matters more than the same issue on a page with 10.

Tips for Better Audits

• Schedule recurring crawls. Set up monthly automated crawls (paid version only) to catch new issues before they compound. Compare crawl data over time to spot trends.
• Crawl staging before launch. Audit your staging environment before pushing major site changes to production. Catch redirect loops, broken templates, and missing meta tags before they affect live rankings.
• Use list mode for targeted audits. Instead of crawling the entire site, paste a specific list of URLs into Screaming Frog (Mode > List). This is useful for auditing a specific section, checking URLs from a migration spreadsheet, or validating fixes.
• Combine with log file analysis. For large sites, supplement Screaming Frog data with server log analysis to understand how Googlebot actually crawls your site versus how it could crawl it. Tools like Screaming Frog Log File Analyser handle this.

Try Screaming Frog →

Troubleshooting Common Issues

Crawl gets stuck or is extremely slow:

Reduce concurrent threads to 1-2 and increase request delay. Check if your server or CDN (Cloudflare, Sucuri) is rate-limiting or blocking the crawler. Whitelist Screaming Frog's user agent in your WAF rules if needed.

JavaScript content not appearing:

Verify that JavaScript rendering is enabled (Configuration > Spider > Rendering > JavaScript). Increase the AJAX timeout under Configuration > Spider > Rendering if pages need more time to load dynamic content. Check the browser console in Screaming Frog's rendered page view for JavaScript errors.

Memory errors on large sites:

Switch to database storage mode (Configuration > System > Storage Mode > Database Storage). This uses disk instead of RAM and can handle millions of URLs, though crawl speed will decrease slightly.

Crawl returns mostly 403 or 429 errors:

Your server or firewall is blocking the crawler. Try switching the user agent to a standard browser, reducing crawl speed, or whitelisting your IP address in the server's firewall configuration.

Pages showing different content than expected:

Check whether your site serves different content based on user agent, cookies, or geographic location. Try switching user agents in Screaming Frog's configuration and compare results.

FAQ

How often should I run a full site audit with Screaming Frog?

Run a comprehensive audit at least once per quarter. For sites that publish content frequently or undergo regular development changes, monthly audits are better. Set up automated scheduled crawls in the paid version to maintain consistency. Additionally, always run an audit before and after major site changes like redesigns, migrations, or CMS updates.

Try Screaming Frog →

Is the free version of Screaming Frog enough for a site audit?

The free version crawls up to 500 URLs, which is sufficient for small sites. However, it lacks key features: you can't save crawl data, connect APIs (Search Console, GA4, PageSpeed), schedule crawls, or use JavaScript rendering. For any site you're seriously managing, the paid license at £259/year pays for itself quickly by catching issues that would otherwise cost organic traffic.

Try Screaming Frog →

How is Screaming Frog different from cloud-based SEO audit tools?

Cloud-based tools like Sitebulb, Ahrefs Site Audit, and Semrush Site Audit run crawls on their servers and present results in a web dashboard. Screaming Frog runs locally on your machine, which means your data stays private, you have full control over crawl settings, and there are no per-project limits. The tradeoff is that local crawling uses your machine's resources and internet connection. Many SEO professionals use Screaming Frog for deep technical audits and cloud tools for ongoing monitoring.

Try Ahrefs →

Try Semrush →

Can Screaming Frog audit JavaScript-heavy single-page applications (SPAs)?

Yes, with JavaScript rendering enabled. Screaming Frog uses an embedded Chromium browser to execute JavaScript and render pages. However, SPAs with client-side routing can be tricky — the crawler may not discover all routes automatically. Use list mode to provide specific URLs, or ensure your SPA has server-side rendering (SSR) or static site generation (SSG) in place, which also helps search engines crawl your content directly.

Try Screaming Frog →

What should I do with the audit results?

Export the findings into a spreadsheet and categorize issues by severity and effort required. Share the document with your development team or client with clear priorities: critical issues first (server errors, broken high-traffic pages), then high-impact optimizations (title tags, redirect chains), then ongoing improvements (alt text, meta descriptions). Track fixes over time by comparing subsequent crawl data. The goal isn't perfection — it's systematic improvement.