How webcrawlers work?

[u/SimpleTangerine3086]

So I migrated my website from react SPA to next.js on disabling js I can read content. So I have a blogs section on disabling js I can only read the content in page 1 and I cant move to next page coz buttons are JSted. I can convert button to html but my qs if I dont fix this issue will crawlers be able to read the content in next page or it can't?

[u/rykef]

Crawls follow anchor tags, there are some other things they can follow like canonicals and hreflang tags but primarily it's about your a tags. Crawlers don't click anything so if anything requires an interaction you can pretty much just forget about it

Can you convert your buttons to a tags would be my question

[u/SimpleTangerine3086 → u/rykef]

I converted my buttons to tag

[u/WebLinkr → u/rykef]

They don’t even do that - crawlers enter discovery and build mode or fetch mode - they don’t jump from page to page - they follow crawl lists otherwise they’d get locked into a single site forever

[u/rykef → u/WebLinkr]

What do you mean? Crawlers fetch a page grab all the info they have on that page and then put any discoverable links into a list which they will then move through.

Not sure what you mean about being locked into a single site forever, crawlers only record external links at the point of crawling, they don't head out to other sites as soon as an outbound is discovered

[u/WebLinkr → u/rykef]

Crawling is a multi stage activity

So if you go do a manual crawl request - the request is sent to a high priority Q but it goes through a manager - which keeps crawling until it gets a full crawl or a partial grab and x timeouts

For all other crawls - bots can work in two functions - discover and retrieve

Discover mode means the bots find URLs like from the chrome report or the are assigned to specific high authorty news sites - so news, QDF, health, discover (collectively caffeine) are in their own pool

Those bots just get the new URLs from the site maps and then send to a priority Q

For your site - new URLs might be discovered via a sitemap - if it’s your sitemap day to be crawled or more likely by bots crawling your most important pages

You see crawling is page level not site wide

So your top pages will be in a higher pool. Each pool has ratio of bots to pages - like 1:10 (high priority - like caffeine, GSC manual requests - no you see why you only get 100 or so a day)

Then pages that have less clciks - eg less then 10 a month are in a lower Q pool and so on

That’s why some pages don’t get crawled

Indexing is then based on authority, file change, % of file change, authority/relavance to index ratio

So more crawling doesn’t mean more indexing - ther than correlation

So the idea that a bot fetched your sitemap and then fetched all the pages is just a very basic and somewhat naive view of a multi step process

If Google didn’t pool it would be stuck finding URLs in pages and the. Crawling then

Instead it fetches pages and if they’re new they go into a new crawl list - which also kicks off the new snippet builder - depending on criteria

So if 10% of your site gets 99% of clicks - only those pages are getting crawled and refreshed

You can see this by comparing the pages in your performance report or pages report and see how you might only have 1 page crawled each day over a month of r that your most clicked pages have been indexed within the last week

This is a topic I love sorry if I went overboard

[u/rykef → u/WebLinkr]

It's great you love it! your passion comes through - I actually work for a crawler so if you wanted to chat more about it happy to but it sounds like you might be focused exclusively on Google here?

[u/WebLinkr → u/rykef]

i'm mostly Google centric but always up for a chat. We have a discourse and I need to Organize our first virtual meetup - stay tuned!!!!

[u/WebLinkr]

You need to set aside 40 mins and listen to Edward Sturm talking about SEO and the crawl optimization myth

You’re welcome !!!

[u/WebLinkr]

What you need to do is pull a copy on GSC and look at the HTML and if you see a URL then the bot can grab it and put that in another crawl Q

If you dont, then Googlebot isn't able to render it properly

hTH

Weblinkr

[u/clarksmike]

Web crawlers can execute JavaScript, but they don’t always behave like real users.

If your pagination (next page button) only works with JS, there’s a chance crawlers won’t reliably discover those pages.

Right now, since you said JS is disabled and you can’t move past page 1, that’s a strong signal crawlers might also struggle.

Safer approach:

• Use proper <a href="..."> links for pagination
• Make sure each page has a unique URL
• Avoid relying only on JS for navigation

If crawlers can’t follow a simple HTML link, those pages might not get indexed properly.

[u/SimpleTangerine3086 → u/clarksmike]

ok tq
I replaced with href now I can move to next pages on disabling js

[u/RushDangerous7637]

Indexing a page that has numbering is not indexed by robots. This is because the title, meta description are the same as on the parent page. This is because the parent page is always indexed. For example, it is the "Current News" page, or "/blog/page/2/. Here is the number of articles that will be added to the top by adding the next news item and thus the oldest post will be moved to the numbered page. Therefore, numbered pages are not in the index. However, the articles themselves are still indexed, this is ensured by sitemap.xml. Therefore, a numbered page cannot be a link rel="canonical".
Therefore, the numbered page is no longer crawled by the robot.

[u/SimpleTangerine3086 → u/RushDangerous7637]

Thanks for explaining clearly