The SEOST Crawling & Indexing Playbook

Crawling & Indexing.

If Google can't crawl it, render it and decide to store it, nothing else you do in SEO matters. Here's how discovery and indexation actually work — and how to unblock the pages that aren't getting in.

Terry Samuels
By Terry SamuelsFounder, SEO Spring Training · Updated October 2026
Crawl budget & crawl paths Robots.txt & sitemaps Canonicals & duplicates Index coverage reports

Discovery and indexation are the first two gates in search. Clear them and your content gets a chance to rank — fail them and the best page on the internet is invisible.

Crawling and indexing is the quiet half of technical SEO, and it's where I find the fastest wins on almost every audit. A noindex left on from staging, a canonical pointing at the wrong URL, a sitemap full of 301s — these cost real pages real rankings, and they take minutes to fix once you can see them.

This hub walks the whole path: how bots find your URLs, how Google decides what to store, and the Search Console reports that show you exactly where pages are falling out. When you want to work through it on a real site, come learn it live at SEO Spring Training.

The Path In

How a page gets into Google.

Every URL passes through three gates before it can rank. Know which gate a page is stuck at and the fix is usually obvious.

01

Discovered

Google finds the URL — from an internal link, an external link, or your sitemap. Orphan pages with no links in often never get found at all.

02

Crawled

Googlebot fetches the page. Robots.txt, server errors, timeouts and redirect chains can stop or waste this step before the content is ever read.

03

Indexed

Google renders, understands and decides whether to store the page. Noindex tags, bad canonicals and thin content get a URL dropped here.

3Gates: discover, crawl, index
2Files Google reads first
8thYear teaching it live
1Room to learn it all
Half the "ranking problems" I'm handed aren't ranking problems at all. The page was never crawled, never indexed, or quietly told Google not to bother.
Terry SamuelsTerry SamuelsFounder & Host, SEO Spring Training

Learn crawling & indexing from the people doing it.

Reading gets you started. The fastest way to level up is four days in the room at SEO Spring Training — real crawls, real Search Console data, fixes you can run Monday.

Get your SEOST pass → 8th Annual · April 7–11, 2027 · Chandler, Arizona · Passes from $897

Questions

Crawling & indexing, answered.

What is the difference between crawling and indexing?

Crawling is when a search engine's bot fetches your page by following links and reading your sitemap. Indexing is the separate decision to render that page, understand it, and store it so it can appear in results. A page can be crawled and still never indexed.

Why isn't my page getting indexed?

The usual causes are a noindex tag, a canonical pointing elsewhere, a robots.txt block, thin or duplicate content, or a page Google judges low value. Check the URL Inspection tool in Search Console — it tells you exactly which stage failed and why.

What is crawl budget and do I need to worry about it?

Crawl budget is roughly how many URLs Google will fetch from your site in a given window. Most sites under a few thousand pages never hit the ceiling. It matters for large e-commerce sites, faceted navigation, or sites wasting crawls on parameter URLs and redirect chains.

How do I tell Google to crawl my site faster?

Submit an accurate XML sitemap, fix internal linking so important pages are close to the home page, remove crawl traps, and keep your server fast. You can request indexing for a single URL in Search Console, but site-wide speed comes from structure and a clean crawl path.

Does blocking a page in robots.txt remove it from Google?

No. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results from external links, just without a snippet. To remove a page, let Google crawl it and return a noindex tag, or use the removals tool for an urgent takedown.

Terry Samuels

About the author

Terry Samuels

Founder and host of SEO Spring Training and the SEOST Digital Marketing Conference — a Salterra event in Chandler, Arizona. Terry has spent years building and ranking real businesses, and brings the practitioners who do the same to one stage each spring.

Last updated October 2026 · Reviewed by the SEOST team
Learn technical SEO live at SEOSTApr 7–11, 2027 · Chandler, AZ · Passes from $897
Get Your Pass →