Advanced Technical SEO Audit · Crawl, Render, Index & Performance

The most common failure mode in technical SEO audits is completeness without prioritisation. An audit that lists 300 issues — from missing H1 tags to server response time to redirect chains — without differentiating between issues that are causing material ranking suppression and issues that are technically present but functionally inconsequential, is not useful to the business.

A practitioner-quality audit organises findings by business impact: how much organic revenue or traffic is this issue costing? The answer requires connecting technical findings to organic performance data — not just listing what is wrong, but quantifying what fixing it is worth. This requires understanding both the technical issue and the organic search landscape for the affected pages.

A crawl analysis using Screaming Frog, Sitebulb, or a similar crawler simulates how Googlebot navigates your site. It reveals: pages that cannot be reached from the site's internal link structure; redirect chains and loops; pages returning error codes (404, 500); pages blocked by robots.txt; and pages with canonicalisation problems.

The crawl configuration decisions that determine audit quality: crawl with JavaScript rendering enabled (to capture what Googlebot sees for JS-rendered content) or disabled (to identify what a raw HTML crawl reveals). Run both for sites with significant JavaScript. Set the crawl to follow the same rules as Googlebot: respect robots.txt, follow rel="nofollow" links, and honour meta robots directives. For very large sites, crawl depth limits and page sampling may be necessary — focus on commercially important sections first.

High-priority crawl issues: orphaned pages (pages not linked from anywhere in the site's internal link graph — Googlebot may not find them); redirect chains longer than 2 hops (each hop adds latency and loses a small percentage of link equity); soft 404s (pages returning 200 status code but containing "page not found" content — confuse Googlebot and waste crawl budget); and pages in the sitemap that return non-200 status codes (sitemap should only contain indexable, live pages).

Indexation issues fall into two categories: pages that should be indexed but are not, and pages that are indexed but should not be. Both have negative SEO consequences — the former means missing organic traffic from pages that should rank; the latter means Googlebot wasting crawl budget on low-value pages and potentially diluting the site's overall quality signal.

Diagnosing under-indexation: compare the number of pages on the site (from crawl) against the number of pages in the Google index (from Search Console's Coverage report and site: operator). A site with 10,000 pages but only 3,000 indexed has a systematic indexation problem. Causes include: robots.txt blocking, noindex meta tags (intentional or accidental), canonical tags pointing away from the page, very thin content that Google classifies as not worth indexing, or crawl budget limitations preventing Googlebot from reaching all pages.

Diagnosing over-indexation: identify URL patterns in Google's index that should not be there — internal search result pages, parameter-generated duplicate URLs, staging/development pages accidentally indexed, user-generated content that meets the thin content threshold. Tools: Screaming Frog custom extraction of canonical and noindex tags; Google Search Console URL inspection; and a crawl of the live index using site: queries with specific URL patterns.

JavaScript-rendered content — where the HTML served by the server is a skeleton and the actual content is injected by JavaScript executing in the browser — creates a two-phase crawl and render process for Googlebot. Googlebot fetches the initial HTML (crawl phase), then queues the page for JavaScript rendering (render phase). The render queue can introduce significant delay — documented as potentially hours or days for very large sites — between when a page is crawled and when its rendered content is available for indexing.

Diagnosing JavaScript SEO issues: compare the raw HTML source (view-source: in browser) with the rendered DOM (browser developer tools, Elements tab). Content visible in the DOM but not in view-source is JavaScript-rendered. Next, use Google Search Console's URL Inspection tool to see the rendered version Google has of the page — if it differs significantly from what you see in your browser, there is a rendering problem.

Solutions by severity: for critical content (body text, H1, links), ensure it is available in the initial HTML response (server-side rendering or static generation) — do not rely on client-side rendering for content you need indexed. For interactive content, consider hybrid rendering: serve the indexable content as static HTML, progressively enhance with JavaScript. For single-page applications, implement proper dynamic rendering or SSR for Googlebot user-agent requests.

Core Web Vitals (CWV) are measured from real user data (Chrome User Experience Report, CrUX) — not from synthetic lab tests. This distinction is critical: PageSpeed Insights scores and Lighthouse scores are lab measurements that predict field performance but do not perfectly correlate with the CrUX data that Google uses for ranking signals. A page can pass Lighthouse but fail CWV at the 75th percentile of real user experience.

LCP (Largest Contentful Paint) — the time until the main content element is visible — is the most commonly failing CWV metric. Advanced LCP diagnosis: identify the LCP element using the Performance tab in Chrome DevTools (it shows the LCP element and when it rendered). Common LCP issues: large hero images without explicit width/height attributes or without fetchpriority="high" hint; render-blocking resources (CSS and JS in the head that block rendering); and slow server response time (TTFB above 800ms systematically delays LCP).

CLS (Cumulative Layout Shift) — visual stability as the page loads — is caused by elements that change position after initial render. Common causes: images without explicit dimensions (browser does not know how much space to reserve); ad slots that expand on load; web fonts that cause layout reflow when they load (FOIT/FOUT). Diagnosing CLS: the Layout Instability API in Chrome DevTools identifies the specific elements causing shift. CSS contain: layout on elements that should not trigger layout reflow can prevent CLS propagation.

INP (Interaction to Next Paint) — the new responsiveness metric replacing FID — measures the delay between user interaction and the next visual update. High INP is almost always caused by long JavaScript tasks blocking the main thread. Chrome DevTools' Performance panel shows main thread activity during interactions — identify long tasks (>50ms) and break them up using scheduling APIs or web workers.

Server log files record every request made to your server, including requests from Googlebot. Log file analysis answers the question that no other SEO tool can: what is Googlebot actually doing on your site, and how does its behaviour compare to what you intended? It is the ground truth beneath all other crawl and indexation analysis.

What to look for in log files: the ratio of Googlebot requests to page count (is Googlebot crawling your most important pages, or spending budget on low-value URLs?); the frequency of recrawl for important pages (are key pages being recrawled regularly, suggesting they are recognised as important?); 404 and 500 errors encountered by Googlebot (a high rate indicates broken links or server instability that is affecting crawl efficiency); and the presence of undesirable URL patterns in Googlebot's requests (parameter-generated URLs, internal search pages, session IDs — signals of crawl waste).

Log file tools: Screaming Frog Log File Analyser, Botify, Oncrawl, and custom analysis using Python/pandas for large files. The raw access logs are typically available from your hosting control panel or CDN provider (Cloudflare, Fastly, AWS CloudFront all provide structured log export). Filter for user-agent strings containing "Googlebot" to isolate Googlebot activity.

Internal link architecture determines how PageRank flows through a site. Pages that receive many internal links from authoritative pages (homepage, category pages) will generally rank better than equally-good pages that are buried in the site hierarchy with few internal links pointing to them. Auditing internal link architecture reveals: pages with zero or very few internal links (orphaned or under-linked pages); pages that have high external link equity (backlinks) but poor internal link distribution from that page to commercially important pages; and depth issues — pages that require 4+ clicks from the homepage are typically under-crawled and under-ranked.

The internal link audit workflow: run a full site crawl and export the internal link graph (source URL, destination URL, anchor text). Count internal links to each page. Identify pages with high organic value (key commercial pages, strong ranking potential) that have fewer internal links than they should. Create a systematic internal linking improvement plan — adding contextual links from high-authority pages to target pages. Anchor text diversity matters: Google documents over-optimised anchor text (too many exact-match keyword anchor links to one page) as a pattern to avoid.

Hreflang is the HTML attribute that tells Google which language/region version of a page to serve to users in different countries. Hreflang errors are extremely common and have material ranking consequences — serving the wrong regional page to a user in a target market, or having hreflang markup errors that cause Google to ignore your international structure entirely.

Critical hreflang implementation requirements: every page with hreflang must include a self-referential hreflang tag (pointing to itself with its own language-region code); all hreflang tags in a set must be reciprocal (if en-gb points to fr-fr, then fr-fr must point back to en-gb); hreflang tags must use valid ISO 639-1 language codes and ISO 3166-1 alpha-2 region codes; and every URL referenced in hreflang tags must return a 200 status code and be accessible by Googlebot.

The most common hreflang errors: missing self-referential tags; non-reciprocal sets (the most frequent error — adding a new language page without updating all existing hreflang sets to include it); hreflang pointing to canonical URLs that are different from the page's actual canonical (a common conflict on e-commerce sites with parameter-based URLs); and x-default misuse (x-default should point to the language-selector page or the most appropriate default page, not just to the homepage in all cases).

Technical SEO fixes should be prioritised by the product of: estimated organic traffic opportunity × fix difficulty (inverse). A major indexation issue affecting 1,000 high-value pages is a higher priority than a structured data implementation that might marginally improve CTR on 50 pages. The prioritisation framework:

Technical SEO audits are commonly presented as long issue lists that engineers and product managers cannot prioritise or act on. A practitioner-quality audit report has: an executive summary stating the most significant findings and estimated organic revenue at risk; a prioritised action list with effort estimates and business impact rationale for each item; clear technical specifications for each fix (not just "fix the canonical tag" but "all parameterised URLs should include a canonical tag pointing to the clean URL, implemented as follows..."); and success metrics (how will you know the fix worked?).

The most effective format for engineering collaboration: individual tickets, not a monolithic report. Convert each P1 and P2 recommendation into a JIRA or Linear ticket with clear acceptance criteria, technical specification, and a link to the relevant section of the audit. Audit findings presented as actionable engineering tickets get fixed; audit findings presented as a PDF get read once and filed.

Further Reading

Go deeper with these reference guides from the Digital Codex library.

Sources & References