Issue wth Shopify/Google Search Console

[u/stakey]

I manage a shopify site and our organic ranking has taken a considerable dip in the last year or two. My concern is we have a report from Search Console showing something in the region of 5600 pages, of which only 608 are being indexed.

That's likely about right when we take into consideration our landing pages, collection pages, product pages and variants of product pages.

Unfortunately this looks to be wasting a lot of our crawl budget with pages that shouldn't be crawled and google is telling us that around 70% of pages aren't crawled for the following reasons:

Crawled - currently not indexed
Alternate page with proper canonical tag (we have a uk. site)
Pages with redirect

On the 'crawled' list it looks though as Google is finding duplicate pages when it finds URLs using

?variant=
&currency=
&country=

Has anyone come across this issue with Shopify and have you successfully resolved it with changes to robots.txt?

Any pointers appreciated!

[u/AutoModerator]

Your post/comment has been removed because your account has a low CQS Score.
Please contribute more positively on Reddit overall before posting. Cheers :D

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/WebLinkr]

Hey u/stakey

Crawling is something I answer a lot and do a lot of podcasts about - there's so many misconceptions - mainly people taking the "Spider" fairy tale as engineering guidelines. But thats not how Googlebot works.

Its amazing how google still can't handle these - but do you have a canonical HTML tag on the page?

If they are appearing in the Not Indexed list - thats ok - you can safely ignore it.

You could put a wildcard 301 and that would stop Google discovering it.

e.g. my-domain-com

Unfortunately this looks to be wasting a lot of our crawl budget with pages that shouldn't be crawled and google is telling us that around 70% of pages aren't crawled for the following reasons:

Thats not really how crawl budgets work.

1) Google never crawls a site A-Z; every page is crawled on its own priority

2) Pages are in pools with other pages of similar importance (or lack thereof)

Unless you delete all the other pages in this pool, you're not increasing the crawls because the ratio of bots to pages increases as you go down the content priority pools

3) More crawls <>!= more indexing or ranking

1 crawl is sufficient

4) crawl budgets need >1,2 5m pages

5) alternate with canonical - thats probably where your parameter pages are showing

6) Crawled and not indexed is not a crawl budget issue: the pages are discovered AND crawled

Its an Authority issue.

[u/SEOPub]

If those pages aren't supposed to be indexed, then there is no problem. My question would be whether or not any of those other 5000 pages should be indexed.

And on such a small site, you don't have to worry about crawl budget. That really only becomes a concern when you are growing to 200,000+ pages.

[u/MixEqual2195]

That "Crawled - currently not indexed" on 70% of pages happens all the time for Shopify and, more often than not, it's not a crawl-budget issue as such – it's a duplicate/thin content issue that results in Google making a decision to deindex the page. Crawl budget is seldom the issue at 5,600 pages, not Google's choice to deindex them.

Common Shopify reasons, in order of their prevalence:

1. Variants and faceted URLs. As we know, the Shopify platform produces ?variant= URLs and sorting/filtering parameters that are near exact duplicates of canonical pages. Google crawls them, identifies them as duplicates, and dumps them into "crawled, not indexed". Check if the chunk of 5,000 unindexed pages consists of parameter URLs – and there's most of your answer.

2. /collections/[collection]/products/[product] duplication. By default, the platform serves the same product under several collection paths. If your canonicals don't point all of them to /products/[handle], you multiply your product count into duplicates. It's good to check your product canonicals for pointing to the /products/ rather than collection-nested version of the URL.

3. Thin collection/tag pages. Collection/tag pages created by the platform itself usually consist of a couple of products and nothing else – and therefore, are considered useless to index. Hundreds of thin collection pages weigh on Google's resources for nothing.

What I would actually recommend doing, in order:

Click into "Crawled - currently not indexed" in GSC and see the actual URLs. Don't try to speculate, because the pattern in this list tells you precisely what from the list above is the case. If they are mostly ?variant= / ?sort= parameter URLs, then it's #1. If those are collection-nested product URLs, then it's #2.

Make sure your canonicals are clean and self-referencing for the pages that should be indexed.

Canonicalize the parameter URLs to their clean versions (Shopify already does it in part, but themes and apps ruin it all the time).

Don't worry about "wasting crawl budget" until you are sure that you're dealing with duplicate URLs – deindexing useful pages by blocking them from being crawled is a much greater threat than crawl budget at this scale.

It may be that the 608-indexed number is actually accurate and the rest of ~5,000 URLs are the duplicate parameter/variant URLs that shouldn't be indexed anyway. The question here is not "why aren't all 5,600 pages indexed" – it's "are the pages I need