Technical SEO Checklist: Crawling, Indexing and Core Web Vitals

By PDS Team

A technical SEO checklist makes sure search engines can crawl your pages, index the right ones and load them quickly for real visitors. The core checks are simple: nothing important is blocked, unwanted pages are kept out of the index the right way, each page has one clear canonical URL and your Core Web Vitals are in good shape.

This guide groups the checks into crawling, indexing and page experience, using Google’s own documentation, so you can work through them in order.

What should a technical SEO checklist cover?

A technical SEO checklist should cover three stages: whether Google can crawl a page, whether it can index it and whether the page gives visitors a good experience.

Content and links matter a great deal, but they cannot help a page that Google cannot reach or chooses not to index. That is why most audits start with the technical layer. It is also the part that breaks quietly, for example when a plugin update adds a noindex tag or a developer leaves a staging rule in robots.txt.

Here is the full checklist at a glance:

If you would rather have a specialist run these checks, our technical SEO services cover crawling, indexing and site speed audits.

How do you check that Google can crawl your site?

Check your robots.txt file and your XML sitemap first, because these two files shape how Google discovers your pages.

robots.txt

Google says robots.txt is used mainly to manage crawler traffic so your server is not overloaded. It is not a way to keep a page out of Google. A URL blocked in robots.txt can still appear in search results, just without a description, if other pages link to it.

Open yourdomain.com/robots.txt and read every Disallow line. Make sure none of them block pages, CSS or JavaScript files you want Google to see.

XML sitemap

• A single sitemap can hold up to 50,000 URLs or 50MB uncompressed. Split larger sitemaps and use a sitemap index.
• Use full, absolute URLs, such as https://www.example.com/page/, not relative paths.
• Google ignores priority and changefreq. It uses lastmod only when the date is consistently accurate.
• Submit the sitemap in Search Console’s Sitemaps report and add a Sitemap: line to robots.txt.

Submitting a sitemap is a hint to Google, not a guarantee that every URL will be crawled.

How do you control indexing with the robots meta tag and X-Robots-Tag?

Use a robots meta tag in the page’s <head> for HTML pages, and the X-Robots-Tag HTTP header for files such as PDFs, images and videos.

The most common rule is noindex, which tells Google not to show the page in search results. For example: <meta name="robots" content="noindex">. To target only Google’s crawler, use name="googlebot" instead of name="robots".

When rules conflict, Google applies the more restrictive one. For a longer look at robots.txt, the meta robots tag and the indexifembedded rule, read our earlier post on the robots tag and X-Robots-Tag.

The mistake to avoid

Do not block a page in robots.txt and add noindex to it at the same time. Google only sees the noindex rule when it crawls the page. If robots.txt stops the crawl, the rule is never read, and the URL may stay in the index. Pages you want removed must stay crawlable.

How do you make sure Google indexes the right version of each page?

Give every piece of content one canonical URL and make your signals agree on it.

Many sites serve the same page at several addresses, for example with and without “www”, with tracking parameters or with filter parameters. Google picks one version as the canonical. Its documentation describes redirects and rel="canonical" as strong signals, and sitemap inclusion as a weak signal.

• Point rel="canonical" to the version you want indexed.
• Do not use robots.txt or noindex to pick a canonical. Google advises against both for this job.
• Do not send mixed signals, such as a canonical pointing to one URL while your sitemap lists another.

Check the mobile version

Google uses the mobile version of a site’s content, crawled with its smartphone agent, for indexing and ranking. Make sure your mobile pages have the same main content, structured data and robots meta tags as desktop. Google also warns against lazy-loading primary content only after a user swipes, clicks or types.

How do you read the Page indexing report in Search Console?

Use the Page indexing report to see which pages are indexed and why others are not, then fix only the reasons that are not intentional.

Google says it is normal for some pages not to be indexed, such as duplicates or pages you have blocked on purpose. What matters is that your important pages are indexed. Common statuses include:

• Crawled – currently not indexed: Google crawled the page but has not indexed it yet. It may be indexed later.
• Discovered – currently not indexed: Google knows the URL but has not crawled it yet.
• Excluded by ‘noindex’ tag: remove the rule if you want the page indexed.
• Duplicate without user-selected canonical: Google picked another URL as canonical.
• Soft 404: the page looks like an error but returns a success code. Return a real 404 or add useful content.

After fixing every instance of an issue, click “Validate fix” so Google rechecks the affected URLs.

How do you check Core Web Vitals?

Check that your pages meet Google’s “good” thresholds for LCP, INP and CLS at the 75th percentile of real visits.

INP replaced First Input Delay (FID) as a Core Web Vital in March 2024, so any checklist or report that still talks about FID is out of date.

Use field data first: the Core Web Vitals report in Search Console and PageSpeed Insights both show real-user data where it is available. Lighthouse is useful for lab testing, but it cannot measure INP and uses Total Blocking Time as a stand-in.

Slow server response often holds back LCP. If your pages are slow before the browser even starts rendering, look at your hosting setup; our web hosting services page explains the options we offer.

Technical SEO Checklist FAQs