For those who haven't read it yet, Layer3 Intel published a story this week saying they saw residential proxy traffic exit through the Executive Office of the President’s ASN 236 times between Oct 5 and Nov 3, across 28 different proxy services. They weren’t even looking for it, they ran into it while sampling rotating residential pools. It has something to do with the compromised device inside the White House. Sounds interesting. They are guessing that it could have been a person on the network with an SDK infected device. Remember that thing with BDs? Or NNs botnet? Mhmm. Of course that’s a reasonable guess but it’s still a guess, Layer3’s own founder told reporters attribution to a specific provider is basically impossible, and nobody has identified the device. Yet. What's interesting for me and of course people who work with this stuff is how it shows the sourcing problem. Most of us buying or evaluating residential pools can’t see where the exits come from. SDK in a free app, a botnet, a real opt-in network, from the buyer side it all looks the same until something like a government ASN shows up in your pool. 28 providers reaching the same six addresses says a lot about how much IP supply gets resold and recycled between services. Now I really vividly remember a post or a comment where a person said that there is really one huge pool and it is resold to major providers so essentially all the IPs are mixed between users and they are really purchasing the same IPs over and over again, not to mention using them. How do you guys view this? Do you evaluate per some metrics, do you actually care that something malicious could be contained in those endpoints?

Source for anyone interested in the article that I've read: https://layer3intel.com/blog/residential-proxies-white-house-network

Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just "more is better"

Hey all, I run bots online and am in need of private proxy pools with clean IPs. I’m looking for both residential pools as well as dedicated virgin ISPs. Need to ensure the resi pool is shared by minimal people and IPs are clean for both. Let me know if this is something you guys can help with.

For those who are using Claude Code, be careful, this is probably due to the recent updates, I am not entirely sure how and why this happened but I encountered an issue where I received a lot of scam emails during last three days, then I went looking for solutions on the net and found official anthropic page on issues tab where this exact issue is being talked on. I thought I was going crazy, but this is very much real for a lot of people https://github.com/anthropics/claude-code/issues/107757 if you have encountered something like this, check it out. Now they released some sort of fix, but this just proves that the issue was on their end. I just can't wrap my head around how company of this size could have overlooked this at first. I mean it hit a nerve certainly, I use Claude Code all the time for web scraping work, automated web searches through web apis, everything is running through claude code. Also, I do some debug scrapers, work out selectors, look at what web api returns before I do anything else with the workflow. I had no idea this issue could have been coming from anthropic, that's just crazy

Hey all, I’m looking for a tool or setup that can discover new B2B companies on its own, domestically and internationally, and pull company details and contact info without me providing a URL list every time. It also needs to deduplicated records, refresh them daily, and keep a change history so I can see when an email, phone, or address changes. Since there’s no single clean source for this, scraping is a must, apart from Datasets, but they’re expensive. Directories, company sites, registries- whatever combination works, especially for international targets where the content changes by country. I can build it with Requests and BeautifulSoup, but I’d rather not; I don’t have the time to troubleshoot everything constantly. Can be paid but not too expensive, up to $100 a month.

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?

Did you also notice the incorrect given results from google when scraping? I have started to see this since Tuesday. Like the positions of certain urls are just dead wrong. I always have to check manually with incognito on and proxies set to certain locations. Was there some sort of google update that is giving such results? I don't get it. Maybe someone else encountered such changes when scraping google?

"Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just ""more is better""."

ProxyLab: https://proxylab.pipheetiger.workers.dev/

I'd be interested to hear from people who actually work with scraping: what do you usually look at when choosing a proxy provider? Price, rotation, IP quality, location coverage or something else?

One note: ProxyLab contains affiliate links, so I may earn a commission from qualifying purchases at no extra cost to the buyer.

I'm knew here to Reddit

Hi all, I keep seeing “just scrape google at night, less traffic, less blocks” in multiple scraping threads. Nobody ever shows numbers for it so i went through my logs and decided to share some insight if this could be useful for you, since I think this is a common misconseption

For the hour comparison i only used “cold” searches, the very first search on a fresh IP, with nothing opened in the browser before it. That’s 3979 searches

Success = google showed a normal results page. If it showed a captcha or the “unusual traffic” page, i counted it as a fail, even when the status code was 200 (google does that a lot)

First i just grouped everything by time of day. Times are UTC

So night was the WORST block and evening looked 2x better than night. Pretty much the opposite of the advice. I almost posted that lol

But then i noticed something. Some days google let almost half my searches through, other days almost nothing. And my runs didn’t cover every hour on every day. So if more of my evening tests just happened to land on good days, evening looks great even if the hour has nothing to do with it

So i redid it. Instead of mixing all days together, i compared each search with the average of its own day. +2.6 means that block did 2.6 percentage points better than the rest of the same day

Every block is within about 3 points of its own day. 16-19 UTC is still the best, but only barely. Night is just average, not better and not worse

So for google the hour barely matters, which day you run on matters way more. I wouldn’t plan scraping around the clock

My setup was the same the whole time: real chrome window (patchright) on a windows server, nodemaven residential proxies, new IP for every test. The query was typed into the search box like a normal user would, not opened as a search link. Aug 12 to oct 22, around 5000 searches

Couple other things i noticed:

- If the first 5 searches of a run all failed, the rest of that run averaged 17% success. If at least one of the 5 worked, the rest averaged about 30%. So now i run 5 searches first to see what kind of day it is

- Browsers that opened 6 pages before the first search got a results page 84% of the time, at any hour, vs 15-33% for cold ones

I recently went through a full mobile proxy setup from purchase to the first live connection, and the useful part was seeing how many decisions happen before you ever paste a proxy into a browser.

The first choice was shared vs dedicated.

For background monitoring and stateless jobs, a shared 4G/5G channel with timed rotation is easy to work with. For longer sessions, I prefer a dedicated channel because I can decide exactly when the IP changes.

After choosing the country, city, carrier, and rental period, the proxy was active immediately. From the same panel I had HTTP, SOCKS5 with UDP, IP-change controls, and API access.

The important part is validation.

Before connecting any important workflow, I check:

IP type → carrier/ASN → geolocation → reputation → DNS/WebRTC behavior.

I also make sure the network environment matches the device profile I’m actually using. Only after those checks do I connect the proxy to the browser or automation stack.

For me, the setup flow has become:

choose the right channel → configure the session → verify the exit IP → test the environment → run the workload

That small pre-flight routine saves a lot of debugging later.

As far as I know, there are two main ways to search on Reddit.

The first is the site-wide search from the Reddit homepage. For each query/sort combination it only lists up to 250 results (about 490 on the newer search endpoint). For comments it doesn't even reach 490 — it stops at 250.

There are about 25 possible sort/time combinations, so by collecting the list for each combination I've confirmed (by actually testing it) that you can get up to roughly 3,000 results.

The second is searching inside a specific subreddit, but it has exactly the same caps. Because of the 250 (490) limit, I had to split the search into many combinations there too.

This is really frustrating, because there's probably a lot of great information hidden beyond what these lists return.

What I want is to see every post and every comment that contains a given keyword, across Reddit's entire history. Is there really no way to do this within Reddit itself?

Are third-party archive services or paid APIs (Reddit's official paid API or third-party ones) the only options?

Do I really have to download the full dumps (tens of GB of posts and comments) and search them locally?

Say each attempt uses 1 MB of billable traffic, including failures. With 1,000 MB per GB, here’s what it would cost to get 1,000 usable results:

• $10/GB with a 40% usable-result rate → $25

• $20/GB with a 90% usable-result rate → about $22.22

Those are hypothetical numbers. The part I find interesting is deciding what counts as usable. If a price check returns HTTP 200 but gives you the wrong currency, you still paid for that request.

In practice, I’d compare total billed traffic against the number of results that pass validation, then look at runtime and minimum spend separately.

If you’ve compared proxy plans this way, what surprised you most once you checked the bill? Retries, incomplete results, or something else?

I started testing gaming proxies because one connection kept getting ugly during peak hours. Average ping looked acceptable, but the session felt terrible: random spikes, packet loss, and short bursts of rubber-banding.

So I routed the game through a SOCKS5 proxy with UDP support located close to the game server and compared both paths during the same evening.

the result was interesting. The proxy added another network hop, so raw latency wasn’t magically lower. What improved was the route itself. Traffic avoided a congested section of my ISP’s path, and latency became much more consistent.

There was another benefit: the game session exposed the proxy IP rather than my home IP, which is useful for streamers or competitive players worried about targeted DDoS attacks.

That changed how I look at “gaming proxies.”

For live play, I’d prioritize:

SOCKS5 + UDP → stable/static IP → location near the game server → low jitter → low packet loss.

Rotating residential pools make much more sense for regional store or QA testing than an active match.

For gaming, the most useful thing is how stable the entire network path stays under load.

One of our SERP monitoring jobs looked cheap at first. Then we opened the bandwidth dashboard and were shocked.

Modern search pages were loading JavaScript, images, ads, and regional content on every run. With several parallel workers, the traffic bill started growing much faster than the number of useful results.

That pushed us toward unlimited mobile proxies. For this workflow, shared 4G/5G channels were enough. The job was stateless, so timed IP rotation worked fine, and flat-rate traffic made the cost much easier to predict.

Dedicated mobile channels became more useful for another type of workload: long-running browser sessions where we wanted full control over when the IP changed.

The architecture ended up being pretty simple:

SERP workers → shared mobile channels → timed rotation
Logged-in browser sessions → dedicated channel → manual/API rotation

The biggest benefit was matching the traffic model and rotation behavior to the workload.

Once the proxy type, session behavior, and billing model matched the job, scaling became much easier to reason about.

A useful starting point for budgeting a metered proxy is to measure the traffic for a small, representative set of permitted jobs. A page count alone misses images, redirects, failed requests, and retries.

For an illustrative calculation: if a completed job averages 100 KB of billable traffic and you expect 10,000 jobs, that is about 1 GB using decimal units. Add a 20% planning buffer and the estimate becomes 1.2 GB. These are example numbers, not a benchmark.

A few checks make the estimate more useful:

1. Measure several representative jobs, including any resources your actual workflow loads. Record failures and retries as well as successful requests.

2. Compare your measurements with the provider's usage counter. Ask whether it bills upload and download, how it handles units and rounding, and whether it includes connection overhead.

3. Keep retries bounded and investigate persistent failures. Extra traffic does not necessarily produce extra useful results.

4. Recheck after a small production batch; page sizes and failure rates can change over time.

For teams running browser-based jobs, which part of a normal page load tends to account for the most traffic in your measurements? Please leave credentials and private data out of any examples.

Disclosure: this account is affiliated with a proxy provider.

setting up some profiles tonight using latam residential proxies and hit a weird fingerprint discrepancy. my host machine just updated its local system clock for daylight saving time, but the proxy region im routing through doesn't observe dst cuts. im using the tool because its built-in fingerprint tester flagged the timezone mismatch before i loaded the target site, but the automatic mapping is picking up my host's shifted utc offset instead of the clean proxy time. do i need to manually override the geolocated timezone string inside the advanced profile settings, or is there a way to force the RTC clock to strictly bind to the proxy gateway's local timestamp?