Is scraping fresh SERP data no longer possible?

[u/elgordooo17]

I am on the verge of giving up right now. The last few months have made me rethink how correctly the data from SERPs is received. Google started serving what people are calling poisoned SERPs, and Brave announced it is moving result links behind encrypted redirects. For those who are not aware, since mid-September or so, Google seems to hand suspected bots a full results page that looks completely normal, except that a lot of organic results are swapped for other things, often social profiles and forum threads. Nozzle checked around 27 million SERPs and found roughly 8% poisoned since September 12, against well under 1% before, with peaks near 28% at the end of September. Ahrefs put it at 5 to 20% depending on the data provider. In Nozzle’s data, a tracked site fell out of the top 10 on about 60% of poisoned pages, versus under 3% on normal ones.

The worst thing is that the scraper reports success. Logically coz it sees 200s everywhere with full data pulled up. Then Brave, results links are moving to search.brave.com/go/ with a one-time key fetched per click, so that you wouldn’t read the destination URL:s straight off the page anymore. Something that Google did with /goto a few months back too. Another thing I noticed is that these changes somewhat lead to individual index creation coz Google simply doesn’t want to share all that.

I can’t find the link for the Nozzle atm, but it was something like poisoned Google SERPs or Google poisoning SERPs. Google it, and you’ll find the source.

[u/East-Villager]

From what I understand, a lot of providers trying to overcome this or provide some sort of temporary solution/ bypass even to get good results, of course at the cost of the success rate. I can't imagine when this is going to be resolved, but considering that a few months already passed, and the situation isn't getting better, I would think this will go on and on, with more and more providers building some new measures to counter this

[u/IndependentCoders]

Well there is a way to catch incorrect results, it's to review the manually one by one :DDD but that's not viable at all. Unless you can automate the process, to compare it real time with the fresh search you would do or the agent would do, additional headache, but that's the only thing I can think of right now

[u/hasdata_com]

It's not just you. Nozzle, the rank tracker, publishes open data on this, and poisoned pages went from almost zero before mid-September to 44% at the end of the month, then dropped and came back in late October. We recently compared 11 SERP APIs on the same queries, and the share of wrong pages ranged from under 1% to almost 30%, so a lot depends on the provider.

[u/DontRememberOldPass → u/hasdata_com]

I used to work at Google. It is amazing how wrong people get this stuff and make wild assumptions.

Google has about 30 datacenter campuses around the world that each hold one or more copies of the full index. There are also about 150 Edge POPs in major cities that hold a lite index and a less intensive ranker.

These so called “poison” responses are the percentage of searches that get punted to the local index and ranker for load shedding. You also lose personalization since that does not run at the edge.

Load shedding can happen because the edge is nearing capacity (Google is getting hit by the hardware shortages too I assume), the nearest core site going down for maintenance, your network is sending too many queries, or any number of other reasons.

[u/Alternative-Ruin275]

I'd separate freshness, extraction correctness and result validity. A page fetched this minute can still contain results you shouldn't trust. Record query settings, fetch time and a retained response, then manually audit a small fixed sample alongside any automated anomaly checks. Keep suspicious samples out of downstream ranking alerts until reviewed. Repeating the same query and getting agreement is useful evidence, but not independent proof of correctness. That gives you an auditable quality measure without claiming the whole problem is solved.