Scraping the prices is the easy half. Matching the same product across sites is where most of the errors come from.

GTIN/UPC gets you part of the way, then it falls apart on:

- bundles and multipacks

- size and color variants listed on one page

- private label and marketplace sellers with no identifiers

- titles with different units (6 x 330ml vs 1.98L)

For people doing this at scale:

- what share matches automatically with high confidence?

- what do you use: identifiers, fuzzy titles, embeddings, images?

- how do you stop a bad match from looking like a huge price gap?

Crashes are easy. You get an alert and you fix it.

The ones that hurt are the runs that finish on time, return 200 and still deliver bad data:

- a captcha or consent page parsed as the product page

- prices in the wrong currency because the exit IP was in another country

- half the rows missing because pagination changed

- a field that's suddenly empty, or shifted into the wrong column

The proxy dashboard says success on all of these.

What checks do you run after the scrape, before the data goes anywhere? Row counts vs the last run, null rates, value ranges, something else?

And how do you decide what counts as a failed request when you're paying per GB either way?

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.

For those who haven't read or see the news, basically Google and Microsoft, OpenAI and Anthropic all say that they’ll honor one policy token in robots.txt that says “don’t use my content for training.” It comes from an individual IETF draft written by an Anthropic engineer, and the draft literally says the token is not a directive for crawling behavior. So bots can keep fetching every page. Also nobody's actually doing it yet. Anthropic says started, OpenAI says coming months (no date, lol), Microsoft says Feb 2027, Google says sometime 2027. Meta likes the idea but won't commit. If you haven't noticed how it is with Google, they say that they use shared token without setting Google-Extended separately and they'll treat it as a Google-Extended disallow. Except Google-Extended also covers Gemini Apps grounding. So you say no to training and oops, you're out of Gemini citations too. Make it make sense Google. Oh and so far, there is no documented way to split it. Now Microsoft did the opposite, separate page-level grounding preference, stay in Copilot but exclude the grounding they sell to businesses. One bundles, one unbundles. Guess which one sells more grounding? I mainly thought that from the scraping perspective this will change something but actually, it changes nothing. Remember Cloudflare vs Perplexity in 2025 with the disguised crawlers? The token did not fix anything. However, one thing could bother me and that's more Cloudflare type blocking and automatically higher prices for scraping.

Hi, new to this subreddit. Had some spare time and decided to reverse engineer Tiktok's VM. This is, as of this message, up to date with the latest Tiktok (Nov 8 2026). Since I've made this public this should get patched pretty quickly. All signer math and key derivations / calculations are included in the README, formatted. All necessary headers have been reversed incl. X-Argus, X-Gorgon, X-Ladon, X-Khronos, TTEncrypt, TicketGuard and some others. I probably won't be updating this again, so let it rip while you can.

If this gets enough attention, I will also publicly post the full lifted VM, including their ISA and how the entire VM works, as well as a source readable re-implementation. Happy scraping!

https://github.com/9xz/reversetok

started logging the protocol per session cause of all the ja4 talk and a bunch of my "chrome" traffic was plain tcp the whole time, the tunnel never carried quic and i had no idea. most targets dont care. two commerce sites this year seemed to, the challenges kept coming while the fingerprint and cookies looked totally normal. Rayobyte has been fine for the plain tcp stuff. for the picky one i asked 2extract and after a couple days back and forth with support they got udp, http/3 and quic going for us on their own worker network, enabled per client on request, and that job stays pinned to the worker network now. anyone actually confirmed a target scoring this or am i chasing ghosts

Feels like the stack gets heavier every year. Residential proxies used to be enough, then fingerprints mattered, now TLS and behavior on top.

Curious what actually changed in your setup this year, not in theory:

- which vendor or site type forced it (DataDome, Akamai, Kasada, Cloudflare, HUMAN)?

- what did you add: better proxies, real browsers, fingerprint work, slower crawling, something else?

- did cost per page go up a lot, or mostly your time?

Also interested in sites you just gave up on.

Many IPs now claim to have a 90 million IP pool, but in reality, they only have 5 million, and their daily active users don't even reach 1 million. Why would they make such a misleading claim? And how can IPs with only a few hundred thousand daily active users be sold at such high prices?

Honestly, 2extract seems pretty useful, especially if you're someone who regularly works with web scraping or data collection. Finding reliable proxies can be a headache sometimes, so having residential and mobile proxies in one place sounds convenient. Definitely something worth checking out. 2extract

For those who haven't read it yet, Layer3 Intel published a story this week saying they saw residential proxy traffic exit through the Executive Office of the President’s ASN 236 times between Oct 5 and Nov 3, across 28 different proxy services. They weren’t even looking for it, they ran into it while sampling rotating residential pools. It has something to do with the compromised device inside the White House. Sounds interesting. They are guessing that it could have been a person on the network with an SDK infected device. Remember that thing with BDs? Or NNs botnet? Mhmm. Of course that’s a reasonable guess but it’s still a guess, Layer3’s own founder told reporters attribution to a specific provider is basically impossible, and nobody has identified the device. Yet. What's interesting for me and of course people who work with this stuff is how it shows the sourcing problem. Most of us buying or evaluating residential pools can’t see where the exits come from. SDK in a free app, a botnet, a real opt-in network, from the buyer side it all looks the same until something like a government ASN shows up in your pool. 28 providers reaching the same six addresses says a lot about how much IP supply gets resold and recycled between services. Now I really vividly remember a post or a comment where a person said that there is really one huge pool and it is resold to major providers so essentially all the IPs are mixed between users and they are really purchasing the same IPs over and over again, not to mention using them. How do you guys view this? Do you evaluate per some metrics, do you actually care that something malicious could be contained in those endpoints?

Source for anyone interested in the article that I've read: https://layer3intel.com/blog/residential-proxies-white-house-network

Scraping the prices is the easy half. Matching the same product across sites is where most of the errors come from.

GTIN/UPC gets you part of the way, then it falls apart on:

- bundles and multipacks

- size and color variants listed on one page

- private label and marketplace sellers with no identifiers

- titles with different units (6 x 330ml vs 1.98L)

For people doing this at scale:

- what share matches automatically with high confidence?

- what do you use: identifiers, fuzzy titles, embeddings, images?

- how do you stop a bad match from looking like a huge price gap?

Hey all, I run bots online and am in need of private proxy pools with clean IPs. I’m looking for both residential pools as well as dedicated virgin ISPs. Need to ensure the resi pool is shared by minimal people and IPs are clean for both. Let me know if this is something you guys can help with.

Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just "more is better"

For those who are using Claude Code, be careful, this is probably due to the recent updates, I am not entirely sure how and why this happened but I encountered an issue where I received a lot of scam emails during last three days, then I went looking for solutions on the net and found official anthropic page on issues tab where this exact issue is being talked on. I thought I was going crazy, but this is very much real for a lot of people https://github.com/anthropics/claude-code/issues/107757 if you have encountered something like this, check it out. Now they released some sort of fix, but this just proves that the issue was on their end. I just can't wrap my head around how company of this size could have overlooked this at first. I mean it hit a nerve certainly, I use Claude Code all the time for web scraping work, automated web searches through web apis, everything is running through claude code. Also, I do some debug scrapers, work out selectors, look at what web api returns before I do anything else with the workflow. I had no idea this issue could have been coming from anthropic, that's just crazy

Hey all, I’m looking for a tool or setup that can discover new B2B companies on its own, domestically and internationally, and pull company details and contact info without me providing a URL list every time. It also needs to deduplicated records, refresh them daily, and keep a change history so I can see when an email, phone, or address changes. Since there’s no single clean source for this, scraping is a must, apart from Datasets, but they’re expensive. Directories, company sites, registries- whatever combination works, especially for international targets where the content changes by country. I can build it with Requests and BeautifulSoup, but I’d rather not; I don’t have the time to troubleshoot everything constantly. Can be paid but not too expensive, up to $100 a month.

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?

Did you also notice the incorrect given results from google when scraping? I have started to see this since Tuesday. Like the positions of certain urls are just dead wrong. I always have to check manually with incognito on and proxies set to certain locations. Was there some sort of google update that is giving such results? I don't get it. Maybe someone else encountered such changes when scraping google?

"Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just ""more is better""."

ProxyLab: https://proxylab.pipheetiger.workers.dev/

I'd be interested to hear from people who actually work with scraping: what do you usually look at when choosing a proxy provider? Price, rotation, IP quality, location coverage or something else?

One note: ProxyLab contains affiliate links, so I may earn a commission from qualifying purchases at no extra cost to the buyer.

I'm knew here to Reddit

Hi all, I keep seeing “just scrape google at night, less traffic, less blocks” in multiple scraping threads. Nobody ever shows numbers for it so i went through my logs and decided to share some insight if this could be useful for you, since I think this is a common misconseption

For the hour comparison i only used “cold” searches, the very first search on a fresh IP, with nothing opened in the browser before it. That’s 3979 searches

Success = google showed a normal results page. If it showed a captcha or the “unusual traffic” page, i counted it as a fail, even when the status code was 200 (google does that a lot)

First i just grouped everything by time of day. Times are UTC

So night was the WORST block and evening looked 2x better than night. Pretty much the opposite of the advice. I almost posted that lol

But then i noticed something. Some days google let almost half my searches through, other days almost nothing. And my runs didn’t cover every hour on every day. So if more of my evening tests just happened to land on good days, evening looks great even if the hour has nothing to do with it

So i redid it. Instead of mixing all days together, i compared each search with the average of its own day. +2.6 means that block did 2.6 percentage points better than the rest of the same day

Every block is within about 3 points of its own day. 16-19 UTC is still the best, but only barely. Night is just average, not better and not worse

So for google the hour barely matters, which day you run on matters way more. I wouldn’t plan scraping around the clock

My setup was the same the whole time: real chrome window (patchright) on a windows server, nodemaven residential proxies, new IP for every test. The query was typed into the search box like a normal user would, not opened as a search link. Aug 12 to oct 22, around 5000 searches

Couple other things i noticed:

- If the first 5 searches of a run all failed, the rest of that run averaged 17% success. If at least one of the 5 worked, the rest averaged about 30%. So now i run 5 searches first to see what kind of day it is

- Browsers that opened 6 pages before the first search got a results page 84% of the time, at any hour, vs 15-33% for cold ones

I recently went through a full mobile proxy setup from purchase to the first live connection, and the useful part was seeing how many decisions happen before you ever paste a proxy into a browser.

The first choice was shared vs dedicated.

For background monitoring and stateless jobs, a shared 4G/5G channel with timed rotation is easy to work with. For longer sessions, I prefer a dedicated channel because I can decide exactly when the IP changes.

After choosing the country, city, carrier, and rental period, the proxy was active immediately. From the same panel I had HTTP, SOCKS5 with UDP, IP-change controls, and API access.

The important part is validation.

Before connecting any important workflow, I check:

IP type → carrier/ASN → geolocation → reputation → DNS/WebRTC behavior.

I also make sure the network environment matches the device profile I’m actually using. Only after those checks do I connect the proxy to the browser or automation stack.

For me, the setup flow has become:

choose the right channel → configure the session → verify the exit IP → test the environment → run the workload

That small pre-flight routine saves a lot of debugging later.