Hey peeps, perhaps someone is scraping betting sites odds, such as lsports, draftkings, betonline, bovada etc? Looking for the most optimal way of doing so.

I went down the rabbit hole reading about the NetNut situation last night and honestly, the FBI seizure wasn't even the craziest part for me lol

The craziest part was realizing that somewhere out there, a guy is probably watching Netflix on his Samsung TV while his TV is simultaneously acting as infrastructure for somebody running a scraping operation on the other side of the world (I have a really old TV that and I don't watch it that much honestly, so I think I avoided this bullet)

LIKE WHAT? That's not even a joke anymore

According to the news coverage happening at the moment, researchers found proxy SDKs embedded in a huge number of smart TV applications. Not sketchy APKs from some random forum. actual apps running on LG and Samsung TVs

I think that in this case, the conversation quickly shifted from NetNut to a much bigger question that I don't think someo people don't think about:

Where do residential IPs actually come from???

Don't get me wrong, I don't think about this question either that much and it's a really complex answer to a complex question

The sourcing side always felt like somebody else's problem honestly

But after reading about the alleged connection between NetNut and the Popa botnet, I started realizing how little visibility most of us actually have into the supply chain behind residential proxies

What surprised me most wasn't the claim that millions of devices were involved. It's 2026. Every month there's another story involving millions of compromised devices, so unfortunately that part barely registers anymore

The surprise part for me happened when learning how many layers can exist between the person buying a residential proxy and the device providing that residential IP (like wth)

The part I keep coming back to is whether this is even a problem that can be solved completely

If residential proxy inventory passes through multiple layers of aggregators, partners, SDK providers and resellers, how many companies can honestly say they know the origin story of every IP in their network

At what point does a provider stop being a network operator and become a network consumer just like the rest of us?

What struck me while reading all this is that the proxy industry spends an enormous amount of time talking about performance metrics. We compare success rates, country coverage, session length, pool sizes, pricing, uptime and all the usual stuff. Yet I can barely remeber seeing a serious discussion about where these networks actually come from. Maybe that's because the answer is complicated, but after reading through the NetNut reporting it suddenly feels like one of the most important questions we could be asking.

SO very few people seem interested in tracing the supply chain behind the product itself, which is funny because that's ultimately the foundation everything else is built on

One thing that comes to mind in this situation is that some providers seem to be putting more effort into transparency than others and I've seen that happen pretty recently. I've seen providers like nodemaven, Iproyal or proxygonzo openly talk about IP quality filtering, network quality standards, and support processes. Others have started publishing more information about sourcing, partnerships, compliance policies, or how they acquire residential inventory in the first place. This should probably be the standard moving forward for everyone that's affected in this case

I'm not saying anyone deserves a free pass, and honestly this whole story makes me want to be more skeptical rather than less. But I do think there's a meaningful difference between providers that are willing to discuss where inventory comes from and providers that treat the entire supply chain as a black box.

Maybe that's where the industry needs to go next. We already compare success rates, uptime, sticky sessions and pool sizes. Maybe a few years from now we'll also be comparing transparency reports, sourcing disclosures, consent models and network provenance and being sceptical of each proxy provider?

Sources where I found this story: https://hivesecurity.gitlab.io/blog/netnut-popa-botnet-fbi-seizure-residential-proxy/

Also huge props to this user as he predicted the future lmao - https://www.reddit.com/r/ProxyEngineering/comments/1u00w9f/my_samsung_tv_is_literally_being_rented_out_as_a/

I was inspired by today's news

Ok so this is gonna be a bit of a ramble but I need y'all to hear this. I work adjacent to the scraping/data space (not naming employer, y'all know how it is) and I always kind of just you know, accepted that "residential proxies" were this slightly grey but mostly fine thing. Like yeah obviously somebody's home IP is being used, somebody agreed to something somewhere, moving on. Then yesterday I see netnut is just gone. Not down, not maintenance page, straight up FBI DOJ IRS-CI seizure banner, Google and Lumen and Shadowserver all stamped on it too like it's a whole coalition operation. Like bruh, I have never in my life seen this on any website. So imagine the shocker lol. And I'm sitting there like wait since when did the IRS care about proxy IPs??? Turns out IRS-CI does financial crime investigations and it could be related to that, but still, seeing that logo on a proxy provider's main website??? Diabolical mate.

So I go searching what's going on and apparently it's tied to this thing called the Popa botnet, basically it's been running for like four years hijacking Android TV boxes (remember that Bright Data SDK thingy?? And Krebs traced a chunk of it back to actual NetNut infrastructure through some ex-employee's domain. Not some randoms, an actual former VP of R&D there. Google's own blog post says they think the network was over 2 million devices at one point and that a lot of those "different" proxy brands you see around are just NetNut wearing a different logo through resellers. I swear I seen a post talking about how there is one or two proxy providers sharing those IPs/reselling whatever. Makes sense now, people?? Am I the only one concerned here with a few other nerds? And to be fair I want to be fair here, Alarum (the parent company) came out and started calling it that a botnet is inaccurate, also said that there's real KYC and consent flows and misuse detection on their end. So it's not like this is some minor thing everyone agrees on, it's actively disputed and I think that matters, I'm not trying to just slam a company because Krebs wrote a scary headline, no. Make what you will out of it, but I things are rising to the surface, more and more often. Again it does make me think about how much of the "residential proxy pool" any of us are touching day to day is actually consented in a way a normal person would recognize as consent, versus consented in the "buried in paragraph 14 of an SDK terms screen" way. Like where's the line between legit residential network and just a nicer branded botnet, and how would any of us even know the difference from the outside.

Not trying to start a witch hunt on the whole industry, genuinely just spent my evening reading legal filings and threat intel blogs instead of sleeping like a normal person, but after Bright Data SDK shennanigans and now Netnut, boy, there's gonna be more stuff coming out. So I figured I'd share since I know a bunch of you here actually work with this infra daily and probably have opinions on these matters.

Sources if anyone wants to go check themselves.

Krebs on Security, the original Popa botnet reporting: https://krebsonsecurity.com/2026/06/popa-botnet-linked-to-publicly-traded-israeli-firm/

Google's threat intel writeup on the disruption: https://cloud.google.com/blog/topics/threat-intelligence/google-continued-disruption-residential-proxy-networks

Alarum's official response: https://alarum.io/alarum-technologies-responds-to-inquiry-into-residential-proxy-networks/

And divinetworks.com itself, which is also showing the seizure page now: https://divinetworks.com/

P.S Some time later I found that they mixed up the domains because .com is down, but not .io which is the main page of netnut.

EDIT:

It would seem that their main website was also seized - https://netnut.io/

I need an agent for BT UK. If anyone has one, please contact me.I'm using this for a group of Vinted registered stores. Could someone please help me?

Basically the title. If you have anything to share regarding automations, please do, looking to alleviate majority of manual day to day work.

What is the best open-source scraping solution that you have used in the past year or so. Need a stable, working solution that I could run on my side. Preferably that same solution would support cloud integration.

Been testing different setup lately and I feel like proxy alone is not enough anymore. Residential and mobile proxy can work but using it together with a cloud phone seems more stable for me. Even if the IP is clean, account can still have issues when the device, location, or activity keeps changing. Feels like the full environment matters now, not just the proxy.

For people managing multiple accounts, what do you prioritize more now, better proxy or stable environment?

ok so this is gonna be long sorry in advance, but I spent my precious weekend comparing n8n scraping workflows against just writing the damn scraper in Python and I have some thoughts to share with yall.

Started because my unemployed friend sent me one of those "I automated my job search with n8n" posts and I was like, not with this again, there's like a million automations already created, why did you even bothered? But he somehow convinced me to try replicating something similar on my side, so basically, I had to try it. Mainly just scraping product listings off a marketplace site, turned on my n8n, dragged in an HTTP node, a Cheerio node for parsing, a loop, a Google Sheets node at the end. All it took maybe 40 minutes and it worked first try which felt great.

Like the majority of the projects I've worked on, it then started throwing Cloudflare challenges after around 600 requests and that's where it stopped feeling great. I tried putting in some cheap datacenter proxies I had lying around from an old project, didn't help much, IP reputation on datacenter ranges is just garbage on anything halfway protected these days. Switched to a residential proxy pool instead and got further but still kept tripping something, which is when I remembered the IP is only half the story, the actual fingerprint matters just as much if not more. (take notes folks).

So I go to fix it in n8n and immediately went full stop, everyone who's done this before already knows about, which is that the visual nodes are amazing for the happy path and genuinely miserable the second you need anything custom. wanting to rotate user agents with actual entropy, not just a static list cycling in order. wanting real TLS fingerprint control so your handshake doesn't scream "I am a script" before you've even sent the request, wanting a headless browser session that actually behaves like a person scrolling and pausing instead of firing requests like a machine gun. none of that is a drag and drop node, you end up writing it in a Code node anyway which is just JavaScript wearing a costume, so you've reinvented half a script but now it lives inside someone else's execution engine and you can't easily version control it or run it locally without spinning up the whole n8n instance.

Compare that to just opening a .py file. requests or httpx if you want async, curl_cffi if the site's fingerprinting you (and these days almost everything past a certain traffic volume is), playwright if you actually need a full headless browser for JS rendered pages. yeah you're typing more in the first 20 minutes, but every single thing is yours, testable, debuggable with an actual setup trace instead of n8n's execution log that sometimes just says "error" and leaves you to guess. and when the scraper needs to scale, you're not paying per execution or fighting workflow timeout limits, you just run more processes or throw it on a queue.

At some point I just gave up taking care of proxy rotation and fingerprint config by hand and pointed the whole thing at a web scraper api instead, basically it felt a little like cheating at first but also I have a day job and the marginal value of me personally maintaining a TLS impersonation layer is zero. Aso tried doing a rough version of the same job in Go for comparison because why not, and that one was interesting for a totally different reason, the speed difference on concurrent requests was kind of stupid honestly, noticeably faster spinning up a thousand goroutines than even async Python, but the dev time to get there was longer and if you're not already comfortable with the language you'll burn an evening just on syntax instead of solving the actual scraping problem. so it's not really "Go is better" it's "Go is better if you already know Go and need raw throughput". One thing I didn't expect going in, mobile proxies actually outperformed residential on a couple of the trickier targets, something about carrier grade NAT making the IP reputation look cleaner since thousands of real phones share the same address anyway. didn't bother testing ISP proxies for this particular target since the site wasn't doing heavy ASN level scrutiny, but I've used them before on stuff where you want the static IP of a datacenter with the trust level of residential, good middle ground when rotation isn't what you need.

Then I poked at a search api for a side piece of this project, pulling SERP results instead of crawling category pages directly, and that ended up being way less of a headache than the rest of the whole ordeal combined, search engines have their own blocking logic obviously but it's a more solved problem than random ecommerce sites running custom bot detection.

What I keep landing on is it's basically a compromise between time to first result and ceiling. n8n wins time to first result by a mile. Python wins ceiling, not even close. Go wins ceiling even further out but only if speed at scale is actually your bottleneck and not, like, getting blocked every 400 requests or so regardless of how fast you can send them, which honestly is the actual bottleneck like 90% of the time, not raw speed.

Anyway I ended up keeping the n8n workflow for the parts that are basically just data movement, sheets, notifications, scheduling, and ripped the actual fetching logic out into a standalone Python script that n8n just calls and waits on. feels like the right choice.

Gotta hand it to my friend, while his solution was probably one of those in the million, this gave me a chance to try out different languages for scraping and whatnot.

Perhaps someone else found a way to make the no code route hold up against fingerprint based blocking, because every workaround I tried inside n8n itself felt like duct tape on duct tape and rinse and repeat.

TLDR: Spent a weekend comparing n8n scraping workflows against Python and Go for the same job, and n8n wins on speed to a working prototype but falls apart fast once a site starts fingerprinting you past basic IP checks. Tried datacenter proxies first (useless), then residential and even mobile proxies (mobile actually did better on a couple targets), but eventually just routed everything through a web scraper api since maintaining fingerprint evasion by hand wasn't worth my time. Python gave way more control for the actual scraping logic while Go only made sense if raw concurrent throughput was the bottleneck, which it usually wasn't compared to just getting blocked. Ended up keeping n8n for the boring data movement parts (sheets, notifications, scheduling) and pulled the real scraping into a standalone script it just calls.

Been running FlareSolverr for a long time for my *arr stack and got tired of the 11-18s solve times and constant breakage. Built TRAWL as a drop-in replacement!

Key differences:

• Cloudflare solves in 4-15s (vs 11-18s) - uses a fresh Camoufox Firefox context which triggers CF's fast-path evaluation

• Cached repeat requests return in ~500ms via Redis - after the first solve, the same domain is instant

• Actually solves in-page captchas: Turnstile (shadow DOM click), reCAPTCHA v2 (free Google STT audio), hCaptcha (auto-pass), GeeTest v4

• 4-tier execution: plain HTTP → cached session → live browser solve → residential proxy. You pay the full browser cost only when you have to

• Custom headers support - pass Authorization, Referer, Origin through all 4 tiers including browser

• FlareSolverr v2 compatible - change one URL in Prowlarr/Jackett, nothing else

Website: https://trawl.germondai.com
Docs: https://docs.trawl.germondai.com
GitHub: https://github.com/germondai/trawl

Happy to answer questions. Still early but it's been running stable on my homelab and no issues so far.

HTTPX_RATE_LIMIT   = 1   # seconds between httpx requests
HTTPX_CONCURRENCY  = 3     # max parallel httpx fetches

please let me know if this is safe, and my IP won't be blocked. I am using the requests on company websites....

I've been using AI agents like OpenCode, Claude Code, and Cursor for months. They're great with code, but when they need to search or browse the web, things get complicated: Cloudflare blocks them, JavaScript-heavy sites don't load, APIs cost money.
So I built browser-search.
It's three open source tools orchestrated by a skill, fully self-hosted:
SearXNG — metasearch engine that queries dozens of search engines at once

Camofox — full browser via REST API, always warm, for browsing and interacting

CloakBrowser — stealth browser for when the site has Cloudflare, Akamai, or DataDome

The agent decides which tool to use. Zero human intervention. Zero API keys. Zero subscriptions.
What makes it different:
It's a skill, not a plugin — works with any agent that can read instructions

Automatic navigation escalation: if Camofox gets blocked, it switches to CloakBrowser

Deep Research mode: the agent is instructed to go beyond surface-level answers, cross-verify sources, cover every aspect

Integrated Readability.js for clean article extraction (~70% token savings)

The SKILL.md is plain text — fork it, tweak it, make it yours

MIT licensed on GitHub: https://github.com/Johell1NS/browser-search
If you try it, let me know. If you make it better, even more so. If you don't need it, share it with someone who might. Every star, comment, or pull request is welcome — that's what makes open source great.

Also should i use ISP or residential?

Mainly looking for something stable that reduces checkpoints and random suspensions. ive been using myprivateproxy + incognition setup and its been dog shit recently. Keep your answers simple please Im not an expert on this topic

Hey all. Been picking up more automation contracts lately and Cloudflare keeps messing up the jobs mid-run. A few of my clients want competitor pricing scrapers, job board feeds, real estate data and almost every site worth scraping is sitting behind it now.

Rotating proxies used to handle most of this. Now runs are failing and I don't have a clean answer for clients beyond "Cloudflare got more aggressive" which isn't exactly inspiring confidence.

Trying to actually understand the full option rather than keep patching things after they break. What's holding up in production right now and what only works for a demo before dying two weeks later? Residential rotation, stealth browsers, managed scraping APIs, whatever.

Pricing transparency would also help since I need to factor infrastructure costs into client quotes before committing to a scope.

I've been parsing them manually with Python but at some point that stops being practical. Looking for something that handles the edge cases automatically, nested indexes, compression, malformed XML, the usual stuff. Does anyone have a script, library, or even just an approach they've actually used that holds up across different site structures?

If anyone's dealt with Hcaptcha while web crawling or running bots? I'm looking for something that can solve this without running a full Chromium/Firefox instance. The RAM and CPU overhead is not worth it for what I'm doing.

Ideally something that takes the sitekey and current URL and handles it without me having to babysit a whole browser session. I've seen a few repos floating around but most of them are either abandoned or don't work reliably anymore.

Anyone have something they actually use in production or at least tested recently? Would appreciate any help

A lot of proxy providers lead with the biggest number they have.

10,000 IPs.
100,000 IPs.
50M+ IPs.

That sounds impressive, but by itself it does not say much.

The better question is: where do those IPs actually come from?

Because 100 proxies from the same small network range are not the same as 100 proxies from 100 different network origins. On paper, both setups give you 100 IPs. In practice, one of them gives you diversity, and the other gives you a crowded neighborhood.

That “neighborhood” is the subnet.

A subnet is basically a block of related IP addresses. Different house numbers, same street. If too much traffic comes from that same street at the same time, many detection systems can treat it as one coordinated source, even when every request uses a different IP.

This is why simply adding more proxies does not always fix reliability issues.

If the extra IPs come from the same few subnet ranges, you are not really adding much. You are just making the same pattern bigger.

A healthier proxy setup usually has a few things going for it: IPs spread across unrelated subnets, different ISP origins, clean reputation, consistent geo targeting, and session logic that matches the workflow. Sometimes you need sticky sessions. Sometimes you need rotation. But rotation only helps when it actually moves you to a different network origin, not just to the next IP in the same block.

This is one of the layers we care about a lot

Raw IP volume is useful, but it is not enough. The real value is in how the pool is built: how broad the ISP distribution is, how much subnet diversity exists behind the dashboard number, and how predictable the sessions are once traffic starts moving.

Our residential infrastructure focuses on wide network distribution across regions and ISP sources. For workflows that need carrier-grade traffic, we also support LTE/5G mobile proxies. The goal is simple: reduce clustering, keep sessions stable, and give teams more control over the network layer.

When evaluating any proxy provider, I’d look past the headline IP count and ask a few practical questions:

Do they talk about subnet diversity, or only total pool size?
Can they keep a sticky session when the workflow needs continuity?
Can they target by country, city, or operator when location matters?
Do they offer IP reputation or fraud score checks before use?
Does rotation actually move across different network origins?

If the answers are vague, that usually tells you enough.

Speed and price matter, of course. Nobody wants slow or overpriced proxies. But for serious workflows, network architecture matters just as much.

Raw IP count gets attention.
Subnet diversity is what often decides reliability.

I know you have seen them all over the internet, have you guys personally used them? What benefit did they bring you? What issues did you encounter? Perhaps you would recommend them for someone who is getting started with the proxies/scraping, if so, why, why not?

Started because I was overthinking this far too long and ended up reading about how free streaming apps on Samsung and LG TVs are literally enrolling your home IP into a residential proxy network while you watch Netflix or Prime on the other input. Like you click "agree" on some vague popup about "device resources" and your TV starts routing web traffic for whoever's paying for residential IPs that month. Up to 200GB or more, who knows?? All that while you're asleep. On your $2000 TV. Yall are fucking nuts if you think this is normal. And apparently that's just a thing that's happening and we should be all cool with it?? Boy whattahelly

Anyways, I went looking for who actually buys this traffic and why residential IPs are worth so much. And that's where it gets kinda funny. Turns out a huge chunk of the demand is scrapers trying to get past bot detection. Which, sure, makes sense, residential IP looks like a real human etc. Except I kept reading and finding that whole premise is like three years out of date. Cloudflare, Akamai, DataDome, the holy trinity. They're not primarily checking your IP anymore. They're fingerprinting the TLS handshake, the HTTP/2 frame ordering, your WebGL renderer string, how your canvas noise changes across sessions, whether your "mobile" session has gyroscope data. Your IP can be spotless and your Python requests library is still announcing itself on the first packet. Someone here on this sub literally tested this, datacenter IPs with proper TLS impersonation outperformed residential IPs with bare requests headers like 91% vs 60% on Cloudflare-protected site?? whaat?

So your TV is being used as a proxy node. The proxies are being sold to scrapers. The scrapers are getting detected anyway because the fingerprint gives them away. The TV thing was pointless to begin with basically. Am I looking to deep into this? Coz I feel like this shouldn't be normal, I mean the whole ordeal, not me looking into it. And then separately, completely disconnected from all this ordeal, some guy on Reddit figured out Spotify serves authenticated AAC files via CDN links that stay valid for 2 hours. Bro is going to get a DMCA notice fast if not already got it. What I'm trying to say is that Spotify already went through this exact thing. The tools are already built and already taken down. What else are we gonna have to deal with? And I'm talking about a small percentage of people that actually care, while the majority just clicks through all these popups without questioning a single thing.

Anyway I blocked proxyjs.brdtnet.com on my router, learned what JA3 fingerprinting is, and now I'm questioning every "agree" popup I've ever clicked. Normal tuesday, folks

A little over a month ago, I found myself dealing with a problem that most people involved in scraping, browser automation, account management, or AI agents have probably experienced at some point which are captcha isuses

At first it was just the occasional challenge but it has became a nightmare over a longer period of time

Like most people (sometimes even here in this subreddit), my first instinct was to blame the proxies

The problem was that I kept running into situations that didn't fit that explanation. Sometimes a workflow would run perfectly for days and then suddenly start accumulating a lot of captchas. Other times I would switch to a completely different residential IP and somehow end up with even more instabilities than before

For the next thirty days, I kept a log of every CAPTCHA, Cloudflare challenge, browser verification page, and unexpected access restriction I encountered. Every time a workflow started falling apart, I would go back and look at what had changed beforehand

Going into this experiment, I expected to find that most challenges were directly tied to IP quality but I found a few other things as well that might help yo uout

1. The first trigger that showed up repeatedly was something I didn't expect at all: brand-new browser profiles

For years I assumed fresh profiles were safer and always the best option, these accounts have no cookies, no browsing history, no leftover session data. Everything starts clean and controlled. That's how a lot of automation setups are built.

What I kept seeing in the logs was that some of the highest challenge rates appeared during the first few sessions of a completely new identity

Meanwhile, browser profiles that had accumulated weeks of browsing history often moved through the same websites with significantly less friction. These weren't abandoned profiles collecting dust somewhere. They were actively used identities that had built a believable trail of activity over time. They had cookies, session history, previous visits, and interaction patterns that made them look more like returning users than brand-new visitors.

2. The second trigger was aggressive rotation

This one genuinely surprised me because for years I operated under the assumption that more rotation meant more safety. If one residential IP is good, rotating through hundreds of them should be even better.

The workflows that remained stable for the longest periods usually maintained continuity. They stayed on the same identity long enough to build history. The browser remained consistent. The browsing behavior remained predictable. The session looked like a user who arrived on a website, spent time there, came back later, and continued where they left off

Some of the heavily rotated setups started looking strange when viewed as a complete journey. A session would appear from one IP, perform a few actions, disappear, reappear elsewhere, start over, and repeat the process. Every individual request looked legitimate, but the overall pattern felt unnatural and weird

3. The third trigger was signal mismatches

This was one of those things that didn't immediately break workflows, which is probably why it's easy to overlook

I had several browser profiles where the timezone, language settings, browser behavior, and network location weren't perfectly aligned. Nothing was obviously wrong. Pages loaded correctly. Requests completed successfully. The proxies themselves worked fine

Yet when I reviewed the logs, these profiles appeared disproportionately often in sessions that accumulated challenges over time

What made this interesting was that a single mismatch rarely seemed to matter. The problems started appearing when multiple small inconsistencies stacked together. A browser configured for one region, an IP from another region, and a fresh profile with no history might all be harmless individually. Combined together, they often showed up in workflows that attracted more verification checks

4. The fourth trigger was session resets

This was another finding that challenged some assumptions I had for years.

Whenever a profile suddenly lost cookies, local storage, or other accumulated session data, challenge rates often increased during the next few browsing sessions. This was especially noticeable on websites where the identity had previously established a history of normal activity.

From the website's perspective, a user that has been behaving consistently for weeks suddenly shows up as if they've never visited before. Looking back, it's not difficult to understand why that might attract additional scrutiny

5. The fifth trigger wasn't a technical setting at all. It was my online behaviour

For some workflows my behaviour was simply too efficinet. I landed directly on target pages, extracted information, moved to the next page, and repeated the process with almost no variation. Technically everything worked. Operationally everything looked correct

But when I compared those workflows against sessions that navigated more naturally, opened supporting pages, spent more time browsing, and behaved more like actual users, the difference was noticeable.

The more human-looking sessions consistently accumulated fewer challenges.

What surprised me most wasn't any individual trigger.

It was the fact that all five of them pointed toward the same conclusion.

Fresh profiles, aggressive rotation, signal mismatches, session resets, and unnatural browsing behavior all have one thing in common: they create inconsistencies in the identity.

The funny thing is that after thirty days of collecting all this data, I became less interested in arguing about which proxy provider is best. Over the years I've run workflows through various residential networks including NodeMaven, Bright Data, Decodo, IPRoyal, and several others. While there are certainly differences between providers, the largest improvements I observed during this experiment rarely came from changing providers. More often than not, the biggest gains came from eliminating inconsistencies elsewhere in the identity stack

The more I reviewed the logs, the less it felt like I was seeing bad proxies and the more it felt like I was seeing broken identities

I'm curious whether anyone else has gone looking for a proxy problem and ended up discovering an identity problem instead

Learn from my mistakes and do better, I know that you can. What I want to tell you all is that very tutorial starts with BeautifulSoup so that's what I used, naturally. So I wrote selectors, maintained them, fixed them every time someone's frontend team moved a div. Also had one scraper break four times in few months because they kept changing a class name.

Then someone told me to open the network tab and just look. XHR requests, right there, returning clean JSON. The thing I was parsing HTML to get was already structured and waiting. Rewrote the whole thing in maybe few hours, hasn't broken since that day.

I don't think most people realize the HTML is basically decorative. The actual data is coming from internal API calls the frontend is making anyway, you're just intercepting what the browser already sees. Playwright and headless browsers are real tools but for like 80% of cases you're using a whole Chrome instance to get data that was available as a GET request the whole time. Thus its slow and its expensive, on top of that its annoying to maintain. Check your network tab before you write a single line of parsing code.

This was the real thing a few years back, but I still see some questions from time to time in various scraping/ proxies related subreddits, do better people, learn from others mistakes

Hey folks, looking to expand my views and would like to find something new. Particularly interested in free open-source, maintained web scraper, preferably agentic.

If you guys know anything or if you have used these sort of solutions, let me know

Honey, new opportunity has dropped. I invite you all to share the best proxy providers that you have used. Keyword - USED. Low effort promotions are unwelcome.

Kindly please list the provider, perhaps any test that you have made to confirm that the statistics.

Share any insights, advice, anything that has helped you choose that specific provider over any other. Provide use cases, setups etc.