What is an Index in SEO?

When you search, a search engine doesn’t scan the live web. It looks up matches in its index and serves them almost instantly. Google says its Search index covers hundreds of billions of web pages and is well over 100,000,000 gigabytes in size, like the index in the back of a book.

Web crawlers, also called “spiders” (Google’s is Googlebot), build the index. Google’s How Search Works documentation describes crawlers that explore the web regularly to find pages, while indexing systems render and analyze each page they fetch, tracking signals from keywords to site freshness.

Crawling, indexing, and ranking

Search happens in three stages, and a page has to clear each stage to reach the next.

• Crawling: crawlers discover pages by following links and reading sitemaps, then fetch their content.
• Indexing: the search engine analyzes each fetched page, including its text, images, and links, and stores what it learns in its index.
• Ranking: at query time, the engine pulls matching pages from the index and orders them by relevance and quality.

Why pages don’t get indexed

Three cases cover most missing pages:

• The page can’t be crawled. A robots.txt block, a server error, or a login wall keeps crawlers out.
• The page carries a noindex rule. A noindex tag or X-Robots-Tag header tells Google to drop the page from search results entirely.
• The page was crawled but not selected. Indexing is never guaranteed, even for pages with no technical problems.

Blocking crawling isn’t a reliable way to keep a page out of search, though. Google can still index and show a URL that robots.txt blocks if other pages link to it, because the crawler never gets to see the noindex rule. To keep a page out of results, do the opposite: let it be crawled and add a noindex rule.

Checking whether your pages are indexed

Search Google for site:yourdomain.com. The results are a sample of the pages Google has indexed from that domain, and adding a keyword, like site:yourdomain.com pricing, narrows the check to one topic. For page-level detail, paste the URL into the URL Inspection tool in Google Search Console, and open the Page indexing report to see which pages weren’t indexed and why. Google’s own guidance says sites with fewer than 500 pages probably don’t need the full report; the site: checks cover them.

Getting your pages indexed

Google finds pages through links from pages it already knows about and through sitemaps, so give it both. Submit an XML sitemap in Search Console, request indexing for individual URLs with the URL Inspection tool, and link every page you care about from somewhere on your own site. Then be patient: crawling takes anywhere from a few days to a few weeks, and requesting a crawl doesn’t guarantee inclusion at all. Google accepts no payment to crawl a site more frequently or rank it higher, so treat any “guaranteed indexing” offer with suspicion.

Other meanings of index

The same word names unrelated things in other technical contexts. On a web server, index.html is the default file returned when a visitor requests a directory, which usually makes it the homepage file. In a database, an index is a lookup structure that speeds up queries. This page covers the search engine sense, the one that decides whether your site can appear in results.