Technical SEO: the complete guide to getting found and crawled

Technical SEO has a branding problem: it sounds like the domain of specialists, and a lot of what gets written about it is a laundry list of 200 checks with no sense of which ones matter. The reality is simpler. Technical SEO is the work that lets search engines and, increasingly, AI crawlers reach your pages, render them, and understand what they're about. If that plumbing leaks, your best content and your hardest-won links are wasted, because the engine never properly sees the page. This guide walks the whole stack in the order that actually matters — reach, read, understand — and is honest about where the real leverage is. Most sites don't need 200 fixes. They need a handful of foundational things to be right, and then to stop fiddling and go write something worth ranking.

Start with how crawling actually works

Before any checklist, understand the loop. A crawler discovers a URL (through links or a sitemap), requests it, receives HTML, possibly renders it to run scripts, and then decides whether and how to index what it found. Every technical SEO problem is a failure somewhere in that loop: a URL that's never discovered, a request that's blocked, a response that's too slow or renders empty, or content the engine can't make sense of. Holding that loop in your head is more useful than memorising checks, because it tells you where to look when a page isn't ranking: is it not being found, not being fetched, not being rendered, or not being understood? Each has a different fix.

Crawlability: the thing most worth checking first

The single most common and most damaging technical problem is content the crawler simply can't reach. A stray rule in robots.txt, a noindex tag left over from staging, pages that live behind a login or a form, orphaned pages with no internal links pointing to them — each quietly removes pages from the index without any visible symptom on the page itself. Before optimising anything, confirm that the pages you care about are actually reachable and indexable. This is unglamorous and it is where a surprising share of lost traffic hides: pages that are perfectly good but that the engine was accidentally told, or structurally prevented, from seeing.

Rendering and the JavaScript trap

Modern sites often build their content with JavaScript in the browser, and this creates a specific, common failure: the crawler requests the page, receives a near-empty HTML shell, and — if it doesn't fully execute the scripts, or does so on a delay — sees little or no content, as we explain in our guide to JavaScript SEO. The page looks perfect in your browser and is invisible to the engine. The fix is to make your important content present in the initial HTML the server sends, through server-side rendering or static generation, rather than relying on client-side scripts the crawler may not run. If you take one thing from the rendering section: check what the crawler actually receives, not what your browser displays.

Site architecture and internal linking

How your pages link to each other does double duty: it helps crawlers discover everything, and it tells the engine which pages matter and how they relate. A flat, logical structure where important pages are a few clicks from the home page and related content is interlinked beats a deep, tangled hierarchy where key pages are buried. Internal links also distribute authority around your site, so a well-linked structure lifts your important pages. Good architecture is one of the highest-leverage, lowest-glamour technical investments: it costs little, it helps every page, and it serves discovery, authority, and understanding at once. Treat your internal linking as a deliberate map, not an afterthought.

XML sitemaps: useful, not magic

An XML sitemap is a list of the URLs you want crawled, handed directly to the engine, and it's genuinely useful for large sites, new sites, and pages that are hard to discover through links, as we cover in our XML sitemaps guide. But it is often misunderstood: a sitemap helps discovery, it does not force indexing or improve rankings, and a sitemap full of low-quality or non-canonical URLs can send mixed signals. Keep it current, include only canonical, indexable URLs you actually want in the index, and treat it as a discovery aid rather than a magic lever. For a small, well-linked site it matters less; for a large or poorly-linked one it can be the difference between pages being found and not.

robots.txt: small file, big blast radius

robots.txt controls what crawlers may request, and because it's a single file with site-wide reach, mistakes in it are among the most damaging and most common technical errors, as we detail in our robots.txt guide. A single overly broad disallow can remove whole sections from crawling; a misunderstanding of the difference between blocking crawling and blocking indexing can leave pages indexed but uncrawlable, or crawlable but deindexed. The practical advice: keep robots.txt minimal, understand exactly what each rule does, never use it to try to hide pages from the index (that's what noindex is for), and double-check it after any site migration, when these errors most often creep in.

Canonical tags and duplicate content

Most sites generate duplicate or near-duplicate URLs without meaning to — through parameters, filters, pagination, print versions, or trailing-slash inconsistencies — and this splits signals and confuses indexing. Canonical tags are how you tell the engine which version is the real one to index and credit, as we explain in our canonical tags guide. Used correctly, they consolidate duplicate variations onto a single authoritative URL. Used carelessly — canonicals that point to the wrong page, conflict with other signals, or are applied inconsistently — they cause the engine to index the wrong page or ignore your hint entirely. Getting canonicalisation right is one of the less visible but more consequential parts of technical SEO on any site with parameters or filters.

Getting pages indexed

Discovery and crawling don't guarantee indexing; the engine still decides whether a page is worth storing, as we cover in our guide to getting indexed by Google. Thin, duplicate, or low-value pages increasingly get crawled and then left out of the index, which is the engine telling you the page doesn't earn its place. The fix for indexing problems is usually not a technical trick but a quality one: make the page genuinely worth indexing, ensure it's not a near-duplicate of something already indexed, and make sure nothing technical (noindex, canonical, block) is contradicting your intent. When good pages won't index, check the technical signals first, then ask honestly whether the page is distinct and valuable enough to deserve it.

Core Web Vitals and page speed

Speed and stability affect both users and crawling: slow sites get crawled less thoroughly and frustrate the people who do arrive, and Core Web Vitals turn that into measurable targets for loading, interactivity, and visual stability, as we cover in our Core Web Vitals guide. The honest framing is that speed is a real but usually modest ranking factor and a large user-experience factor. Don't expect a speed improvement alone to transform rankings, but do treat egregious slowness as a genuine problem worth fixing, because it caps crawling, hurts conversions, and sits at the threshold where it can tip close rankings. Fix the worst offenders, then move on rather than chasing a perfect score.

Structured data done honestly

Structured data (schema markup) describes your content to the engine in a machine-readable form, and when it accurately reflects what's genuinely on the page, it helps the engine understand you and can earn richer result listings. The rule is accuracy: markup that describes real content and real information is helpful; markup that claims things not visible on the page, or games rich-result eligibility, is a guidelines violation that can get your markup ignored or penalised. Add the structured data that genuinely applies to your content — articles, products, FAQs, organisations — and keep it honest. It's a clarity tool, not a ranking hack, and treating it as the latter is how sites get their rich results pulled.

Technical SEO for AI crawlers

The technical foundations increasingly serve a second audience: the crawlers behind AI answer engines, which need the same access, rendering, and clarity to include your content in their answers. A site that's crawlable, renders its content server-side, and is cleanly structured is one both Google and AI assistants can use; a site that hides content behind scripts or blocks is invisible to both. This convergence is good news: the technical work you do for classic SEO largely prepares you for the AI answer layer too, so you're not maintaining two separate technical setups. Getting the fundamentals right is increasingly a single investment that serves every way people now find you.

How to prioritise without drowning

The trap in technical SEO is treating every check as equally urgent and drowning in a 200-item audit. Prioritise by the crawl loop and by blast radius. First, anything that stops pages being reached or indexed at all — robots blocks, noindex mistakes, rendering failures, orphaned key pages — because those cap everything. Then the signal-clarity items — canonicals, sitemaps, structured data — that affect how well you're understood. Then the optimisation items — speed, Core Web Vitals — that offer real but smaller gains. Most sites get the majority of the available benefit from the first group, which is usually a short list. Fix those, verify, and resist the urge to keep polishing details that don't move anything.

How to audit your own site

A practical audit follows the same order. Check what the crawler receives for your key pages (view the rendered and raw HTML, confirm content is present). Check index coverage in your search console for pages missing or wrongly excluded. Check robots.txt and for stray noindex tags. Map your internal linking and find orphaned important pages. Review canonicalisation on any section with parameters or duplicates. Then check Core Web Vitals and fix the worst. Doing this once properly, and re-checking after any migration or redesign (when technical regressions love to appear), keeps the plumbing sound. You don't need to do it weekly; you need to do it well, and then watch for the changes that break things.

The bottom line

Technical SEO is the plumbing that lets engines and AI crawlers reach, render, and understand your pages, and when it leaks, your content and links are wasted. Work it in the order that matters: first make sure your pages can be reached and indexed at all (crawlability, robots, rendering, architecture), then make your signals clear (canonicals, sitemaps, structured data), then optimise speed and stability. Most of the available benefit lives in that first group, which is usually a short list, so prioritise by what stops pages being seen rather than chasing a perfect audit score. Get the foundations genuinely solid, verify what the crawler actually receives, re-check after migrations, and then put your energy where it compounds — into content and authority worth ranking.

Frequently asked questions

What is technical SEO, in plain terms?

The work that lets search engines and AI crawlers reach your pages, render them, and understand what they're about — the plumbing beneath content and links. It covers crawling and indexing, rendering and JavaScript, site architecture, sitemaps, robots.txt, canonicals, speed, and structured data. If the plumbing leaks, even great content and strong links are wasted because the engine never properly sees the page.

What's the most important technical SEO issue to check first?

Crawlability — whether the engine can actually reach and index your important pages. A stray robots.txt rule, a leftover noindex tag, pages behind logins, or orphaned pages with no internal links all quietly remove content from the index with no visible symptom on the page. It's unglamorous, but it's where a surprising share of lost traffic hides, so confirm your key pages are reachable before optimising anything else.

Why does my page look fine but not rank or get indexed?

Often it's a rendering problem: if your content is built by JavaScript in the browser and the crawler receives a near-empty HTML shell, it may see little or no content even though the page looks perfect to you. Check what the crawler actually receives (the raw and rendered HTML), not what your browser displays. The fix is making important content present in the initial server HTML via server-side rendering or static generation.

Does site speed actually affect rankings?

It's a real but usually modest ranking factor and a large user-experience and crawling factor. Don't expect a speed improvement alone to transform rankings, but egregious slowness is worth fixing because it caps how thoroughly you're crawled, hurts conversions, and can tip close rankings. Fix the worst Core Web Vitals offenders, then move on — chasing a perfect score past that point rarely moves anything.

How do I prioritise technical SEO without drowning in a 200-point audit?

Prioritise by the crawl loop and blast radius. First, anything that stops pages being reached or indexed at all (robots blocks, noindex mistakes, rendering failures, orphaned key pages) — these cap everything. Then signal-clarity items (canonicals, sitemaps, structured data). Then optimisation items (speed, Core Web Vitals) for real but smaller gains. Most sites get the majority of the benefit from that first, usually short, list.