Content wins rankings, but only after the machinery works. Before Google can judge what a page says, it has to find the page, fetch it, read it, store it and serve it to a visitor whose phone renders it properly. Any break in that chain is invisible in the writing and decisive in the results — which is why a technical audit belongs at the start of every year, every redesign and every migration.
Key Takeaways
- Google’s minimum for indexing is three things: Googlebot is not blocked, the page returns HTTP 200, and the page has indexable content. Meeting them still does not guarantee indexing.
- robots.txt controls crawling, not indexing. A blocked page can still appear in search results; use
noindexto keep a page out. - A sitemap is a hint, not an order: one file holds at most 50,000 URLs or 50 MB, and lists canonical URLs only.
- A canonical tag is a strong signal, not a rule. Redirects are stronger; sitemap inclusion is weaker.
- Good Core Web Vitals mean LCP of 2.5 seconds or less, INP of 200 milliseconds or less and CLS of 0.1 or less, at the 75th percentile of real visits.
- AI-crawler controls are three separate switches: Google-Extended for Gemini training, the Search Console control for AI Overviews and AI Mode, and robots.txt rules for other bots.
What Is Technical SEO, and Why Does It Come First?
Technical SEO is the work that makes your site findable, readable and usable by search engines: crawling, rendering, indexing and page experience. It comes first because nothing else works without it — a page Google cannot reach cannot rank, however good its content, and a page that loads badly loses the visitor it finally wins.
Think of it as plumbing: readers never see it working, and they notice immediately when it fails. Google’s own framing is equally practical. Its technical requirements for indexing are just three conditions — Googlebot is not blocked, the page works, and the page has indexable content — and the page is blunt that meeting them does not guarantee indexing. The checklist below is about meeting those conditions deliberately, then checking the rest of the chain.
Can Google Crawl Your Site?
Crawlability is whether Googlebot can reach your pages at all, and it fails more often from small oversights than from big decisions. Start with the file Google reads first: your robots.txt, at yoursite.com/robots.txt, which tells crawlers which URLs they can access on your site.
robots.txt exists mainly to avoid overloading your site with requests — it is not a mechanism for keeping a web page out of Google. Check three things: no important page is disallowed — a leftover Disallow: / from staging is the classic launch disaster; the CSS and JavaScript your pages need are not blocked, since Google says pages that need them should keep them open; and every important page returns HTTP 200, because Google only indexes pages served with a success status.
Is Your Sitemap Actually Helping Google?
An XML sitemap is a list of the URLs you want in search results, and its job is to help Googlebot crawl your site more intelligently — not to force it. Google is direct about the limit: submitting a sitemap is merely a hint, and it does not guarantee that Google will download the sitemap or use it for crawling.
The rules come from Google’s sitemap documentation. One sitemap holds at most 50,000 URLs or 50 MB uncompressed; larger sites split it and use a sitemap index. Use absolute URLs in the sitemap. List only the canonical URLs you actually want in search: no redirects, no noindex pages, no 404s. Google ignores <priority> and <changefreq>; it does use <lastmod>, but only if consistently accurate — a wrong date is worse than none. Submit it in Search Console’s Sitemaps report.
Are Your Pages Indexed — or Just Published?
Publishing a page and having it indexed are two different events, and the gap between them is where most “my content doesn’t rank” complaints actually live. Open Search Console’s Page indexing report: it lists every URL Google knows about and why each one is not indexed, which is the fastest way to separate a crawling problem from an indexing decision.
The exclusion reasons are your work list. “Crawled — currently not indexed” usually means Google looked and chose not to store the page, often for low-quality content. That is a content verdict: improve or remove the page — our guide to handling thin content in the AI search era covers the logic. “Alternate page with proper canonical tag” is not an error — Google picked a different URL as the canonical. And remember: noindex only works if Googlebot can crawl the page, so never combine it with a robots.txt block. Mobile-first indexing has applied to all sites since 31 October 2023, so content hidden on the mobile page effectively does not exist for search.
Which URL Is the Real One?
One page on your site can live at half a dozen addresses — HTTP and HTTPS, www and bare domain, tracking parameters on the end. Google groups near-identical pages into a cluster and picks the most complete and useful as the canonical, and your job is to make that choice obvious.
Google ranks the methods by strength in its canonicalization documentation: redirects are a strong signal, rel="canonical" annotations are a strong signal, and sitemap inclusion is a weak one. The methods stack when they agree — so be consistent: one canonical URL, named everywhere. Use the right tool: a canonical tag is for duplicates that must stay live — our canonical tags guide covers the detail; a 301 redirect is for a duplicate that should stop existing — our redirect strategy guide covers the mechanics; noindex is for a page that should not appear in search at all.
How Fast Is Fast Enough?
Core Web Vitals are Google’s three measures of how a page feels to real visitors, and they have exact pass marks. A page passes when LCP is 2.5 seconds or less, INP is 200 milliseconds or less, and CLS is 0.1 or less, each measured at the 75th percentile of real page loads, separately for mobile and desktop.
The measurement matters as much as the numbers: the scores come from field data — real Chrome users over the previous 28 days — not from a single lab test. PageSpeed Insights shows field data alongside lab results; Lighthouse cannot measure INP. Fix by metric: LCP is usually a slow server response, an unoptimised hero image, or render-blocking resources — compress the largest visible element and keep it out of lazy-loading; INP is usually heavy JavaScript blocking the main thread; CLS is usually images without reserved dimensions or fonts that swap late. INP replaced First Input Delay (FID) on 12 March 2024. Google keeps this in proportion: Core Web Vitals are used by its ranking systems, but there is no single signal, and Google says it will show the most relevant content even if the page experience is sub-par.
Does Google See the Same Page Your Visitors See?
A page can look perfect in your browser and broken to Googlebot, and the difference is usually JavaScript or mobile rendering. Google does run JavaScript when it crawls, but with differences and limitations — and content that only appears after a visitor interacts with the page may never be seen at all.
The practical rule is simple: anything essential must be in the HTML that arrives with the page, not assembled by scripts after load — the main text, the navigation links Googlebot follows, and the canonical tag, which Google says to put in the HTML source and keep JavaScript from changing. If a page depends on JavaScript, test it with the URL Inspection tool’s live test. Check mobile rendering too: open key pages on a real phone and look for tap targets too close together and text needing horizontal scrolling.
Is Your Structured Data Earning Anything?
Structured data is markup that helps Google understand what a page is — a product, an article, an event, a recipe — and it can earn a page an enhanced appearance in results. It cannot earn a page a ranking it does not deserve, so first check whether the markup you have is still doing anything at all.
Validate before you celebrate: run pages through Google’s Rich Results Test, and read Search Console’s enhancements reports, which list errors site-wide rather than one URL at a time. Keep the markup honest: it must describe content actually visible on the page — Google’s spam policies treat misrepresenting markup as a violation. One change to absorb: FAQ rich results no longer appear in Google Search — Google’s changelog records they would stop appearing from 7 May 2026, with the documentation removed the following month. FAQPage markup is still valid schema.org, and Google says there is no need to proactively remove it — but do not add it expecting expandable questions under your result.
Is Your Site Secure Enough for Search?
HTTPS has been a ranking signal since 2014, and by 2026 the check is less about having a certificate than about having it implemented without gaps. Every page should be served over HTTPS, every internal link and canonical should point at the HTTPS version, and the HTTP version should redirect to it permanently.
The gaps are where sites fail. Mixed content — an HTTPS page loading images, scripts or styles over HTTP — triggers browser warnings and can break the page’s security indicator. An expired certificate, or one issued for the wrong host, is worse than none: Google’s canonical logic will then prefer the HTTP version strongly. Then implement HSTS, which tells browsers to only ever use HTTPS for your domain. Keep the certificate renewing automatically; an expired certificate is a site-down event in slow motion, and the fix is automation, not vigilance.
Do You Need to Worry About Crawl Budget?
Crawl budget is the number of pages Googlebot will fetch from your site, and for most sites the honest answer is no. Google’s guide is written for three kinds of site: those with more than a million unique pages, those with more than 10,000 pages that change daily, and those whose new pages are not being crawled promptly.
Google’s test: if your new pages tend to be crawled the same day they are published, you do not need the guide. For everyone else, its advice is the same short list — keep the sitemap up to date and check the Page indexing report regularly. Googlebot’s attention is finite: every fetch spent on a duplicate, a parameter variant or a dead URL is a fetch not spent on a new page. If your site is large enough for the question to matter, check your server logs: a log full of 404s, redirect chains and parameter URLs is a site spending its budget on nothing.
How Do You Control AI Crawlers Without Hurting Search?
AI crawlers are the newest item on the technical checklist, and the costliest to get wrong, because the controls look alike and do different things. There are three separate switches, and flipping the wrong one can remove you from AI answers you wanted — or leave your content in training data you meant to exclude.
First, Google-Extended: a token in robots.txt controlling whether content Google crawls may train future Gemini models and ground Gemini Apps answers. Google is explicit — it is a product token, not a crawler, it issues no requests, and it does not affect inclusion in Google Search or ranking. Second, Search Console’s Search generative AI control, under Settings: it excludes the whole site from AI Overviews, AI Mode and generative AI features in Discover, rolled out worldwide on 31 August 2026. It does not affect AI training — for that, Google points you back to Google-Extended. Third, robots.txt rules for other AI companies’ crawlers: blocking a training crawler such as GPTBot opts you out of model training, while blocking a search crawler such as OAI-SearchBot removes you from ChatGPT search answers entirely. Our guide to blocking AI crawlers without losing AI search visibility has the full token list.
The Checklist
- robots.txt does not block important pages, CSS or JavaScript, and lists your sitemap.
- Every important page returns HTTP 200 and has at least one internal link.
- An XML sitemap of canonical, indexable, absolute HTTPS URLs is submitted.
- Page indexing report has no unexpected excluded pages;
noindexpages are not blocked by robots.txt. - Every indexable page has a self-referencing canonical with an absolute URL; redirects, sitemaps and links name the same URL.
- LCP ≤2.5s, INP ≤200ms, CLS ≤0.1 at the 75th percentile.
- Mobile shows the same content as desktop; key content is in the HTML, not behind JavaScript.
- Structured data validates in the Rich Results Test; no FAQ schema for a rich result that no longer exists.
- HTTPS everywhere, no mixed content, valid certificate, HSTS enabled, renewals automated.
- AI-crawler stance is deliberate, not accidental.
Frequently Asked Questions
How often should I run a technical SEO audit?
At least once a quarter, and after every launch, redesign or migration. A monthly glance at the Page indexing and Core Web Vitals reports catches most problems early.
Does technical SEO matter for a small site?
Yes, and the list is shorter, not different. A small site still needs crawlable pages, a clean sitemap, canonicals that agree, HTTPS and fast loading. Skip only crawl-budget analysis, which Google reserves for very large sites.
Will fixing Core Web Vitals improve my rankings?
It can help, but it will not rescue weak content. Google says Core Web Vitals are used by its ranking systems, yet it will still show the most relevant content even if the page experience is sub-par.
Should I block AI crawlers from my site?
Only with a deliberate stance, not a blanket rule. Blocking training crawlers such as GPTBot opts you out of model training; blocking search crawlers such as OAI-SearchBot removes you from that engine’s answers entirely.
What is the single most common technical SEO mistake?
A staging block left on after launch — a Disallow: / in robots.txt or a site-wide noindex nobody removed. It hides the whole site at once.
The Bottom Line
Technical SEO is maintenance, not a project with a finish line. The sites that keep their rankings are the ones whose plumbing gets checked on a schedule — robots.txt after every launch, the Page indexing report every month, Core Web Vitals after every template change. Run the checklist, fix what it finds, and let the content do the rest.

