The SEOST Crawling & Indexing Playbook
Crawling & Indexing.
If Google can't crawl it, render it and decide to store it, nothing else you do in SEO matters. Here's how discovery and indexation actually work — and how to unblock the pages that aren't getting in.
Discovery and indexation are the first two gates in search. Clear them and your content gets a chance to rank — fail them and the best page on the internet is invisible.
Crawling and indexing is the quiet half of technical SEO, and it's where I find the fastest wins on almost every audit. A noindex left on from staging, a canonical pointing at the wrong URL, a sitemap full of 301s — these cost real pages real rankings, and they take minutes to fix once you can see them.
This hub walks the whole path: how bots find your URLs, how Google decides what to store, and the Search Console reports that show you exactly where pages are falling out. When you want to work through it on a real site, come learn it live at SEO Spring Training.
The Moving Parts
Six parts of crawling & indexing.
Discovery isn't one setting — it's a chain. Each link below is its own deep-dive, with guides and sibling hubs underneath.
Robots.txt
The first file every crawler reads — what to allow, what to block, and the mistakes that hide whole sites by accident.
Explore →XML Sitemaps
The map you hand Google. How to build one that only lists canonical, indexable URLs — and keep it clean.
Explore →Crawl Budget
How many URLs Google will fetch, why parameter traps and redirect chains waste it, and when it actually matters.
Explore →Canonicalization
Duplicate URLs, parameters and pagination — telling Google which version is the real one so ranking signals consolidate.
Explore →Index Coverage
Reading the Search Console Pages report — crawled-not-indexed, discovered-not-indexed, and how to act on each.
Explore →Rendering & JS
Content that only appears after JavaScript runs has its own crawl path. See the JavaScript SEO hub for the full story.
Explore →The Path In
How a page gets into Google.
Every URL passes through three gates before it can rank. Know which gate a page is stuck at and the fix is usually obvious.
Discovered
Google finds the URL — from an internal link, an external link, or your sitemap. Orphan pages with no links in often never get found at all.
Crawled
Googlebot fetches the page. Robots.txt, server errors, timeouts and redirect chains can stop or waste this step before the content is ever read.
Indexed
Google renders, understands and decides whether to store the page. Noindex tags, bad canonicals and thin content get a URL dropped here.
Start Here
Crawling & indexing guides.
Practical, start-to-finish walkthroughs — the discovery and indexation moves we teach, written down.
The robots.txt guide
What the file does, the exact syntax, and the blocks that quietly hide pages you wanted indexed.
XML sitemaps for SEO
How to build, split and submit a sitemap that speeds discovery instead of confusing it.
The technical SEO audit checklist
Everything to check — crawl, index, speed, structure — before you spend a dollar on content.
The ecommerce SEO guide
Faceted navigation and crawl budget at scale — the indexation traps that sink online stores.
JavaScript rendering & SEO
When content loads after JavaScript runs, crawling gets a second stage. Here's what Google does with it.
Technical SEO: the full guide
Step back up to the parent pillar — how crawling and indexing fit the whole technical foundation.
Half the "ranking problems" I'm handed aren't ranking problems at all. The page was never crawled, never indexed, or quietly told Google not to bother.
Terry SamuelsFounder & Host, SEO Spring Training
Learn crawling & indexing from the people doing it.
Reading gets you started. The fastest way to level up is four days in the room at SEO Spring Training — real crawls, real Search Console data, fixes you can run Monday.
Questions
Crawling & indexing, answered.
What is the difference between crawling and indexing?
Crawling is when a search engine's bot fetches your page by following links and reading your sitemap. Indexing is the separate decision to render that page, understand it, and store it so it can appear in results. A page can be crawled and still never indexed.
Why isn't my page getting indexed?
The usual causes are a noindex tag, a canonical pointing elsewhere, a robots.txt block, thin or duplicate content, or a page Google judges low value. Check the URL Inspection tool in Search Console — it tells you exactly which stage failed and why.
What is crawl budget and do I need to worry about it?
Crawl budget is roughly how many URLs Google will fetch from your site in a given window. Most sites under a few thousand pages never hit the ceiling. It matters for large e-commerce sites, faceted navigation, or sites wasting crawls on parameter URLs and redirect chains.
How do I tell Google to crawl my site faster?
Submit an accurate XML sitemap, fix internal linking so important pages are close to the home page, remove crawl traps, and keep your server fast. You can request indexing for a single URL in Search Console, but site-wide speed comes from structure and a clean crawl path.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results from external links, just without a snippet. To remove a page, let Google crawl it and return a noindex tag, or use the removals tool for an urgent takedown.
About the author
Terry Samuels
Founder and host of SEO Spring Training and the SEOST Digital Marketing Conference — a Salterra event in Chandler, Arizona. Terry has spent years building and ranking real businesses, and brings the practitioners who do the same to one stage each spring.
Last updated October 2026 · Reviewed by the SEOST team