Real fixes from seven live sites. No pitch.Log in / My reports
Idea to Traffic Pressure Test

Do Thousands of Not Indexed Pages Hurt Your Rankings?

Not indexed URLs cannot rank, so they do not drag your real pages down directly, and large counts of redirects, 404s and noindexed pages are normal. They cost you when Googlebot wastes its visits on endless junk URLs or when near duplicates split signals with your real pages. Find the URL patterns in Search Console, stop generating them, then use 404s, canonicals, noindex and robots.txt in that order.

A common shock in Search Console: you open Indexing > Pages and the gray "Not indexed" bar dwarfs the green one. One site owner on r/SEO this week found 1.58 million not indexed URLs on a business site that has roughly 200 real pages. The worry is always the same: is all that junk dragging down the pages that matter?

The short version: a URL that is not in the index cannot rank, so it cannot pull your other pages down through ranking signals directly. The real costs are indirect, and they depend on what those URLs are and how many of them Googlebot keeps fetching. Here is how to tell which case you are in and what to do about it.

What "not indexed" actually means

The Page indexing report lumps very different things under one gray bar. Click the bar and look at the "Why pages aren't indexed" table. The reasons fall into three groups:

  • Working as intended. "Page with redirect", "Alternate page with proper canonical tag", "Excluded by 'noindex' tag" and "Not found (404)" usually mean you told Google not to index something, or it no longer exists. Large counts here are normal on any site that has changed over time.
  • Google chose not to. "Crawled, currently not indexed" and "Duplicate without user-selected canonical" mean Google fetched the page and decided it was not worth keeping, or picked a different URL as the main version.
  • Google has not gotten to it. "Discovered, currently not indexed" means Google knows the URL exists but has not crawled it yet. On a big pile of junk URLs this is often the largest line.

The first group is harmless. The second and third groups are where you should spend time, because they tell you something about how Google sees the site.

When not indexed pages actually cost you

There are three real ways a mountain of unwanted URLs can hurt, and none of them is "Google subtracts points for each one".

1. Crawl waste. Google's own crawl budget documentation says most sites do not need to worry about crawl budget. It names the cases that do: very large sites (about 1 million or more unique pages) whose content changes weekly, medium sites (about 10,000 or more pages) whose content changes daily, and sites where a large share of URLs sit in "Discovered, currently not indexed". That last case is exactly the 200 real pages and 1.5 million junk URLs situation. If Googlebot spends its visits fetching endless filter, search, calendar or session URLs, your new and updated real pages wait longer to be crawled.

2. Diluted signals. If the junk URLs are near copies of real pages (the same product reached through ten parameter combinations, print versions, tracking parameters), internal links and external links can split across versions, and Google may pick a canonical you did not want.

3. Quality signals on what does get indexed. Pages that are not indexed are not part of how Google judges your site's content. But a site that generates junk at scale rarely keeps all of it out of the index. If thin auto-generated pages are slipping into the index, those indexed pages are the ones that can affect how your site is assessed overall.

If your not indexed URLs are mostly old redirects and 404s, and your real pages are indexed and crawled promptly after updates, you can stop worrying. A big gray bar alone is not a problem.

Find where the junk comes from in 20 minutes

You do not need to look at 1.5 million rows. You need the patterns.

  1. In Search Console go to Indexing > Pages, click "Discovered, currently not indexed" and "Crawled, currently not indexed", and export each list (top right, Export). The export holds a sample of up to 1,000 URLs, which is plenty to spot patterns.
  2. Open the export and sort by URL. Look for repeating shapes: ?sort=, ?filter=, ?page= going into the thousands, /search?q=, /tag/, calendar URLs that run into future years, session IDs, or an old CMS path that still answers.
  3. Go to Settings > Crawl stats and open the report. Check "By response" and "By file type", then click a row to see example URLs Googlebot actually requested. If most requests go to parameter URLs, that is your crawl waste.
  4. Test a few of the URLs yourself. A junk URL that returns 200 with a near copy of a real page is the most expensive kind:
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "https://example.com/products?sort=price&color=red&page=48"

If that prints 200 for a URL that should not exist, your server is generating pages on demand for any parameter. That is the usual source of six and seven figure counts.

Fix it in the right order

Pick the fix by what the URL is. Using the wrong tool can make things worse, so follow this order.

  • Stop generating the URLs. The best fix is upstream. Remove crawlable links to infinite filter combinations, cap pagination, stop the calendar from linking to empty future months, and drop session IDs from URLs. If links stop pointing at junk, Google stops discovering more of it.
  • Return real errors for things that do not exist. A URL with no content should return 404 or 410, not a 200 page that says "no results". Google drops these over time on its own.
  • Canonical for near duplicates you must keep. Sorted or tracked versions of a real page should carry <link rel="canonical" href="https://example.com/products"> pointing at the clean URL.
  • noindex for pages users need but search does not. Internal search results and thin tag pages can use <meta name="robots" content="noindex">. Google has to crawl a page to see noindex, so do not also block it in robots.txt.
  • robots.txt for pure crawl waste. Once the junk is out of the index, block patterns Googlebot should never fetch, such as:
User-agent: *
Disallow: /*?sort=
Disallow: /*?sessionid=
Disallow: /search

Robots.txt stops crawling, not indexing. A blocked URL that is linked from elsewhere can still appear in results without a description, which is why noindex or a 404 comes first and robots.txt second.

Finally, make sure your XML sitemap lists only your real, indexable, canonical pages. For a 200 page site that sitemap should hold about 200 URLs, and that number is the one to compare against the green "Indexed" count.

What to expect after the cleanup

The gray bar shrinks slowly. Google recrawls junk URLs on its own schedule, and on a site with a million of them it can take months for the counts to fall. That is fine. The numbers that tell you the fix worked come sooner:

  • In Crawl stats, a growing share of requests going to your real pages.
  • New or updated pages getting crawled within days, which you can confirm with URL Inspection and its "Last crawl" date.
  • Your real pages showing as indexed with the canonical you chose ("User-declared canonical" and "Google-selected canonical" match in URL Inspection).

Do not use the Removals tool for this. It hides URLs from results for about six months and does nothing to stop crawling or fix the cause.

If you want a quick read on the structural basics first (status codes, canonicals, robots.txt, sitemap and noindex on your main pages), the free site checker runs 30 checks on any URL in about ten seconds. For a site with a long messy history, the Search Console export and Crawl stats steps above are what will actually show you the patterns.

Where this question came from

Questions about not indexed pages and crawl budget

Should I delete pages that are not indexed?

Only if they have no value to visitors. Pages that redirect, return 404 or carry noindex on purpose can stay as they are. Junk URLs generated by filters or search should stop being generated and return 404 or 410.

Does blocking URLs in robots.txt remove them from Google?

No. Robots.txt stops crawling, not indexing, and Google cannot see a noindex tag on a page it is not allowed to fetch. Use noindex or a 404 first, then block crawl waste patterns once they are out of the index.

How long until the not indexed count goes down?

It depends on how often Google recrawls those URLs. On a site with hundreds of thousands of junk URLs it can take months. Judge progress by Crawl stats and how fast your real pages get crawled, not by the gray bar.

Is crawl budget a problem for a small site?

Usually not. Google says crawl budget mainly matters for very large or fast changing sites, or sites where many URLs sit in Discovered, currently not indexed. A small site with a few hundred clean URLs rarely needs to think about it.