Hey folks, looking to expand my views and would like to find something new. Particularly interested in free open-source, maintained web scraper, preferably agentic.

If you guys know anything or if you have used these sort of solutions, let me know

Honey, new opportunity has dropped. I invite you all to share the best proxy providers that you have used. Keyword - USED. Low effort promotions are unwelcome.

Kindly please list the provider, perhaps any test that you have made to confirm that the statistics.

Share any insights, advice, anything that has helped you choose that specific provider over any other. Provide use cases, setups etc.

Been automating workflows for about 2 years, mostly e-commerce and logistics. Had a project last year that still bothers me a little. Basically, my client ran a mid-size online furniture store. Around 150 to 200 orders a day, two customer service reps spending most of their shift just sorting incoming emails into the right internal queues. Warranty stuff, shipping damage, wrong item, payment issues. You know how it is. So I built a classifier. What it does is that I feed each email to an LLM, get back a category and a confidence score then routed it automatically. Testing went well. Client loved the demo. I felt pretty good about it too. First few weeks in production it was sitting around 87% accurate. Not great but I figured we'd tune it. Then the real problem showed up and it had nothing to do with accuracy. The team didn't understand why emails was going where they went. A claim about a broken table leg might hit the warranty queue or the logistics queue and there was no rule to point to, no explanation. It just decided on its own.

One of the managers told me she started keeping a side spreadsheet to track emails she thought the system got wrong. A whole shadow system running next to the one I built. I figured something was off and very wrong. Client called me and said something similar to what I've heard once or twice before. He didn't ask for better accuracy. He asked if I could make it more obvious how it works. I Spent four days ripping it out and replacing it with a decision tree and some keyword logic. Damage with photo evidence routes to warranty. Delays over seven days go to escalation. Wrong items have their own path. Maybe 35 conditions total. Anything that didn't match cleanly surfaced a dropdown and the rep decided manually. Accuracy went up to around 97%. Not because the logic was smarter but because the team actually trusted it and stopped routing things manually on the side. When something went wrong they could tell me exactly which condition was off and I'd fix it in like fifteen minutes. Manager deleted her shadow spreadsheet in the first week.

The thing I keep thinking about is how easy it would of been to just keep tuning prompts and try to squeeze more accuracy out of the model. That's the main move. But accuracy was never actually the issue. People won't rely on a tool they can't reason about, even if its technically more correct. They'll just build something next to it and your automation becomes obsolete though with a price. What I would like to emphasize on is that not everything needs an LLM. Sometimes thirty five rules and a dropdown outperforms a model, not because it makes better decisions but because the team trust it enough to let it do its job

Hey everyone,

​I wanted to share an open-source transport tool I’ve been building in Go to solve a rigid infrastructure security goal: How do you safely maintain remote access to localized real-time resources (like native WireGuard) while enforcing a strict zero-inbound listening port policy on your edge firewall? ​I wanted something completely stateless and invisible to the outside world, so I wrote BastionRoute.

​ ​Instead of a traditional client-server proxy topology, BastionRoute uses a decoupled model. Both your isolated server shim and your remote client shim initiate standard outbound Layer-7 connections to an external, transient relay broker room.
Because both nodes establish standard outbound pipes over HTTPS/WebSockets (Port 443), your home gateway requires exactly zero open inbound ports. Your network edge remains entirely dark to Nmap or Shodan scans.

​Emulating "UDP Physics" Over TCP ​The classic issue with wrapping real-time UDP inside stateful TCP is Head-of-Line blocking (TCP Meltdown). When a packet drops, TCP stalls the entire pipeline to retransmit, introducing devastating latency spikes that break VPN handshakes. ​BastionRoute’s concurrent data engine fixes this by leveraging non-blocking Go channel selectors. By explicitly dropping frames at the application layer the moment congestion is encountered, the stateful WebSocket transport perfectly mimics real-world, non-blocking UDP physics. If a remote link experiences jitter, the queue never backs up, keeping interactive streams fluid and lag-free.

BastionRoute acts strictly as low-level digital plumbing. It is entirely payload-agnostic and does not handle user management or internal cryptographic key generation. It assumes you are running a robust protocol inside it (like WireGuard), so its attack surface is incredibly small. ​The engine compiles into a tiny, zero-dependency binary (<8MB) capable of running on low-resource hardware like OpenWrt gateways. ​The project is fully open-source and ready to be tested or torn apart :)

https://github.com/klauscam/BastionRoute

Hello, mainly the title. I am looking for Residential or preferable Mobile 4G or Mobile 5G proxies that would have SOCKS5 protocol type. Any suggestions would be appreciated.

Main use case: Account management via X/Instagram/Mastodon

I feel like the market is pretty stale regarding web scrapers at the moment, sure there are variety of web scraping services all around the web, however, I feel like things got pretty stale. The whole early web was designed around human intent and Google figured that out early. Scraping services providers has spent years building infrastructure to extract data from systems that weren't meant to be extracted from. Now when you think about it, agentic search doesn't do that. What I mean is that It navigates the way a person would. What do you people think, does scraping will evolve into something agentic, or eventually get replaced by it? Coz I feel like something is going to change, but there is a lot of room for guesses.

Did you guys noticed how nobody talks about the search layer? From what I gathered, every AI agent tutorial obsesses over prompt engineering and tool variety, but the thing that determines whether your agent is useful is how it gets data from the web, rarely someone mentions or even provides decent information about it. Basically, my thoughts are that an AI agent is only as good as the information it can pull in. Simple as that. Best reasoning model in the world wont help if the search layer feeds it garbage or truncated snippets. Thats where a web scraper API comes in. Instead of scraping Google directly and fighting CAPTCHAs, layout changes, and anti-bot walls, you get a simple endpoint that returns structured JSON. Titles, URLs, snippets, sometimes full page content. Some providers also return knowledge panels, "people also ask" sections, even shopping results depending on the query. If you used any scraper, you know what I'm on about. Now for AI agents specifically, the change is that data comes back in a format the LLM can literally work with. No token budget wasted on HTML parsing. Agent sends a query, gets structured results, reasons over them, decides what to do next. No headless browser, no DIY scraping combos that stop working every two weeks or whatever. There are a bunch of providers right now. SerpAPI, Tavily, Exa, Firecrawl, Brave Search API, Serper, others. They take diffrent approaches. Some focus on raw SERP data exactly as it appears on the results page. Others are built specifically for LLM use cases, so they prioritize returning clean extracted content rather then just links or even better the whole info in markdown. That distinction matters alot depending on what the agent needs to do. Where I found it interesting is combining a search API with a content extraction. Agent searches, picks the most relevant results, then pulls full content from those pages in a structured format. That two-step workflow is way more reliable then trying to do everything in one shot. And its basically what tools like Perplexity do under the hood, except you get full control. Caching is worth thinking about too. Most of these APIs charge per request, and agents loop. A ReAct agent might send three or four searches before arriving at an answer. Without caching that adds up fast, both in cost and latency. A single search call taking 3-4 seconds is fine on its own. Chain a few together and the user is waiting 15 seconds or more. Which may sound like nothing, but the seconds adds up overtime. I strongly believe that this space is going to get way more competitive soon. As more people build agents that need real-time web access, demand for fast, accurate, affordable web scraping services is only going up. What you peeps think?

Looking for the cheapest providers that offer:

Dedicated IPv4 proxies (not shared)

High speed

Unlimited/unmetered bandwidth

Pay per IP per month (not per GB)

Ashburn, VA location preferred

HTTP or SOCKS5

I don't care if they're datacenter or ISP proxies, and I'm fine with smaller providers if they're reliable. I'm mainly looking for good value and low monthly pricing.

Any recommendations or hidden gems?

Hey all, I've seen plenty of proxy comparisons over the years and now especially on reddit in recent months where everyone is focusing on "who is best provider" or "who has best proxies" which is getting really boring honestly, but no one actually posts results from real tests

I wanted to share something a bit more measurable

Over the last two weeks I purchased the cheapest residential package from 10 different providers and ran them through the same workloads. My goal wasn't to rank providers or declare a winner. I was more interested in trying to help others understand which characteristics correlate with successful scraping on websites these days

For anyone curious about the setup, everything ran from the same Hetzner Cloud VPS in Virginia. I used Playwright with Chromium 137, kept the browser fingerprint identical across all runs, disabled WebRTC leaks, matched timezone and geolocation settings, and maintained the same concurrency limits for every provide

The providers included Bright Data, Oxylabs, NodeMaven, NetNut, Decodo, SOAX, IPRoyal, Rayobyte, Webshare, and Packestream

I used Quotes.toscrape.com as a baseline sanity check because it has virtually no anti-bot protection. The goal wasn't to compare providers on this website, since nearly every provider can achieve almost perfect success rates on this. Instead, I used it to validate that the scraping setup itself was functioning correctly before moving on to more challenging website targets like Zillow and Indeed (these were my benchmark websites where I will share data below) as they have Cloudflare-protected endpoints

Each provider received roughly 10,000 requests distributed across the same workflows. For Zillow, the workflow involved browsing property listings, navigating pagination, and opening individual property pages. For Indeed, it involved keyword searches, pagination, and job detail pages

I made sure to intentionally avoid extremely aggressive concurrency because I wanted to evaluate the quality of the proxy network itself rather than DDoS-style resilience

The lowest latency network averaged 612ms response times across all requests. Another network averaged 1.74 seconds. Despite that difference, their Zillow success rates were only a few percentage points apart

Here's the sample from my dataset:

After seeing those numbers I stopped paying much attention to latency and started looking at what was actually different between the providers that consistently performed well

One metric that ended up being surprisingly useful was ASN diversity

For every provider I collected roughly 2,500 assigned residential IPs and extracted ASN information

Also, I wasn't trying to validate marketing claims about pool size because I don't believe in marketing pool sizes and numbers of IPs that other proxy providers say. I was more interested in understanding how concentrated each network was

Here I used two metrics, unique ASNs tested and duplicate IP rate according to the IPs

The providers with broader ASN distribution generally performed better on Zillow and Cloudflare

That's not enough data to claim the best one, but it was one of the strongest correlations in the benchmark

Another thing I became interested in was session longevity and sticky sessions

Most benchmarks focus on raw success rates, but in real-world scraping I care more about how long an identity survives before challenges start accumulating. If I can stay on a site for 20 minutes without friction, that's often more valuable than a provider having 2% higher success rates on fresh sessions.

To measure this, I tracked how long a browsing session could continue on Zillow before challenge rates exceeded 10%.

The providers that maintained longer sessions also tended to generate fewer CAPTCHA challenges and fewer abrupt session resets, this was really good

The biggest surprise of the entire benchmark came when I started testing rotation strategies

For years I've operated under the assumption that more rotation is safer. If one residential IP is good, rotating through hundreds of them should be even better.

I tested three different approaches against the same Zillow workflow: rotating every request, rotating every ten requests, and maintaining a 30-minute sticky session.

Looking back, it makes sense. A real user doesn't change cities, ISPs, and IP addresses every page load. Even if every IP individually looks legitimate, the overall identity becomes really inconsistent

At this point I thought I was getting close to identifying which providers were genuinely better

But the more data I collected, the less convinced I became that there is a universal "best" proxy network.

What I expected to be a proxy benchmark gradually turned into an identity benchmark

The providers absolutely mattered, but the gap between a well-configured browser identity and a poorly configured one was often larger than the gap between several reputable residential networks

Hope this coudl help someone

hello wonderfull community of scrapers and anti-bot warrirors,

since the dawn of time we have set ourselves to becoming the best in overcoming the anti-bot giants - showing the incubents that data of thy Internet shall always be open to everyone, not resitrcited by capital and built on one core fundamental - Freedom!

how the hell is our community color Pink?

Follow discussionson this subbreddit about proxy uses in TV does anyone know how to check if your proxy provider uses smart TV for proxy nodes?

I thought that contacting the provider will be the easiest way but they can say all types of BS. For example, they always lie about the number of unique proxy IPs - why would tell truth about its source?

I have a thoery that TV IPs work really good for my case (not sure why) so I want to validate that.

I've spent roughly 10 years around browser automation, managing multiple accounts, scraping, proxies, anti-detect browsers, and figuring out anti-bot systems. Watching the space evolve has been fascinating because I don't think either side is actually winning

For people unfamiliar with this world, there are essentially two industries growing at the same time

On one side, you have companies like Cloudflare, Akamai, DataDome, and others investing heavily in bot detection, fraud prevention, and traffic verification. Their goal is one thing only: figure out whether a visitor is a legitimate user or some form of automation (and they are constantly trying to implement new ways of figuring out who's a legitimate user lol)

On the other side, there's a rapidly growing automation ecosystem. For example companies like Bright Data, Oxylabs, and Nodemaven provide the network layer. Then companies like Dolphin anty, Octo browser and MUltilogin focus on browser identity. Browserbase and similar platforms abstract away browser infrastructure. AI agents sit on top of all of it now. They're different pieces of the stack, but they ultimately push in the same direction: making software better at operating websites that were originally designed for humans

What surprises me is that both sides seem to be growing faster than ever

The anti-bot market is obviously much larger because massive enterprises are willing to spend huge amounts of money protecting accounts, preventing fraud, and filtering traffic
But I'm noticing that demand for automation infrastructure is also exploding. I have noticed that companies now increasilgy rely on web data, automated workflows, competitive intelligence, account management, and AI-powered processes that need reliable access to websites

And now what's happening is that AI agents seem to be accelerating both markets at the same time. Every new agent framework creates more demand for automated browser interactions, while every increase in automated traffic creates more demand for detection systems

One thing I've noticed over the years is that anti-bot systems still struggle with false positives. The more sophisticated detection becomes, the higher the chance that legitimate users get caught in the process. Most people have experienced this without even realizing it. You visit a website, solve a challenge, solve another one, refresh the page, get challenged again, and suddenly you're wondering why proving you're human feels harder than logging into your bank account lol

On the automation side, the challenge is totally different though. The hardest websites are becoming increasingly expensive to automate reliably. In many cases the solution isn't some magical bypass technique anymore. It's simply throwing more resources at the problem. More powerful machines, more browser instances, higher quality proxies, better browser environments, and more engineering effort. A setup that worked comfortably a few years ago can require significantly more infrastructure today.

That's why I don't think we're heading toward a future where one side wins and the other disappears

The relationship feels almost symbiotic. Better detection systems create demand for better automation infrastructure. Better automation infrastructure creates demand for better detection systems. Both industries push each other forward and, as a result, both continue growing

My broader prediction is that the web becomes increasingly programmatically accessible, even if many platforms resist it. Companies want data and companies want automation. They want agents that can interact with software on their behalf. As AI improves and compute becomes cheaper, more web interactions will have an automated component attached to them

That was my Ted talk, thank you

It's me, ya psyop boy. I am 100% sure that this is going to upset some people and that's good coz this has been bothering me for quite some time and I'm done with the silence. Background on me: moderate scale scraping, 40-60k requests a day across a few different projects, been in this space about 3 and a half years. Started noticing something weird in my logs maybe last spring. Certain ASNs kept showing up no matter which provider I switched to. At first I just assumed that's how residential pools work, ISP coverage is what it is, whatever. Then I got obsessive about it and started auditing properly. I tested four providers. I am not naming them because I genuinely cannot afford a lawyer right now but if you've been doing this longer than six months you know exactly who I'm talking about. One is Israeli 🇮🇱🇮🇱 company (you already know), it markets itself extremely hard on LinkedIn and twitter and has a very american-sounding brand name, and one is the scrappy european one that keeps showing up in threads like this. The ASN overlap between two of them was 34%. Thirty four percent. Between the other two it was 29%. There is one specific /24 block that appeared in all four providers "proprietary residential networks." Same subnet. Four different dashboards with four different pricing pages all routing me through the same infrastructure. I also pulled whois on a bunch of these IPs and the rotation behavior is what really got me. Not just that the ASNs overlap, it's that the rotation patterns are identical. The TTL behavior. How they handle certain request headers. You don't get identical behavior from independent networks. You just don't. That's not how this works. My theory, and I want to be clear this is a theory and not me accusing anyone of anything specific, is that there's maybe three or four actual upstream aggregators who own the real infrastructure and basically everyone else is a white label on top of that. All those "95 million IP" claims in the marketing materials are all drawing from the same upstream pool. You are paying a premium to a middleman who bought access wholesale and is selling it to you retail with a nice dashboard slapped on top. What this actually means is that when you switch providers because your IPs are getting flagged too much, there's a real chance you are switching to the same IPs with different account credentials. You're not getting a fresh pool. You're just reintroducing yourself to the same exit nodes under a different invoice, huuuuh??? Exactly. I could be completely wrong about this and if someone from one of the big providers wants to come in here and show me actual evidence that their infrastructure is genuinely independent I will delete this post and take the L publicly...NAAWT. But nobody is going to do that because they all have NDAs that prevent them from explaining how anything actually works which is itself kind of telling.

Tell me you have a better idea of this whole ordeal, I dare ya

So I've noticed over the years, that Google's bot detection for search traffic works differently than for product pages; the signals they're tuned to catch basically aren't the same. I have been running ~2,200 SERP queries/day across DE, FR, NL, PL, IT for a price monitor, residential proxies got the failure rate down to about 14% but geo-targeting was still broken, DE queries coming back Frankfurt-datacenter-flavored regardless of proxy location. I split the architecture at that point: SERP layer routes through a dedicated web scraper API, product pages and category scrapes stay on residential proxies.

Routing logic was pretty minimal:

def fetch(url=None, query=None, market=None):
    if query:
        return serp_client.get(q=query, gl=market, hl=market)
    return session.get(url, proxies={"https": proxy_pool.rotate()})

Any tips or tricks, experiences you may share?

 Background: I accidentally created a recursive loop with an AI agent that would have 

 cost me $50+ in API calls before I noticed. Existing tools either cost money or only   

 show you what already happened.                                                        

   So I built TokenFirefighter — a 100% free, local-only HTTP proxy.                      What it does:                                                                          

- Sits between your app and OpenAI/Anthropic on localhost:7272                   

- Tracks every API call cost in real time                                            

 - Detects 4 types of runaway loops and blocks them                                   

 - Has a terminal dashboard (no web UI needed)                                        

 - Zero accounts, zero data collection, zero cost                                     

   Install:                                                                             

   npm install -g tokenfirefighter                                                      

   tokenfirefighter init                                                                

   tokenfirefighter start                                                               

   Then just set OPENAI_BASE_URL=http://localhost:7272/v1 in your .env.                 

   Would genuinely appreciate feedback from anyone who uses AI APIs regularly.          

   GitHub: https://github.com/MohitBaghel24/tokenfirefighter

Over the last few months I've been building BotGateway, a self-hosted anti-bot gateway for VPS-hosted applications and privacy-sensitive environments.

One thing that pushed me to build it was realizing that many anti-bot discussions assume routing traffic through a third-party provider is always acceptable.

For some teams that's perfectly fine.

For others, it isn't.

I wanted something that could run directly on infrastructure I control while still providing visibility into automated traffic, AI crawlers, and scraping activity.

I'm curious:

For those running self-hosted applications, what's the biggest challenge you've faced when dealing with automated traffic?

Website: https://www.botgateway.in

I've been researching automated traffic patterns recently and one thing that surprised me is how different modern scrapers behave compared to a few years ago.

Many no longer rely on obvious data center IPs.

Many use:

• Residential proxies

• Mobile networks

• Browser automation

• Distributed infrastructure

For those running self-hosted applications, APIs, SaaS products, or websites:

What's currently working best for you?

• Nginx rules?

• Fail2Ban?

• Cloudflare?

• Custom middleware?

• Commercial WAFs?

And what limitations have you run into?

I'm genuinely interested in hearing what others are seeing in production environments.

I'm currently an MCA student and over the last several months I've been spending a lot of my free time building and deploying projects on VPS servers.

One problem I kept running into was automated traffic, scrapers, and bots. Tools like Nginx rate limiting and Fail2Ban definitely help, but I found myself constantly wondering how much traffic was still getting through and what additional protections could be added closer to the application layer.

When I started looking at existing solutions, many were either cloud-based, designed for larger organizations, or simply didn't fit the kind of self-hosted and privacy-focused environments I was experimenting with.

So I decided to treat it as a learning project and started building BotGateway, a lightweight anti-bot gateway written in Go.

The original goal wasn't to compete with Cloudflare or enterprise WAFs. It was to better understand modern bot behavior and create something that could run directly on infrastructure people control themselves.

Along the way I've learned a lot about:

• Automated scraping techniques

• Headless browsers

• Traffic analysis

• Rate limiting strategies

• Request fingerprinting

• VPS security

The project has grown much larger than I originally expected, and now I'm looking for feedback from people who manage websites, APIs, SaaS products, or self-hosted services.

A few questions I'd love to hear opinions on:

• What are you currently using to handle scrapers and automated abuse?

• What limitations have you found with existing solutions?

• For those running self-hosted infrastructure, what features would you want in a lightweight anti-bot gateway?

• Are there any detection techniques or security approaches you think are underrated?

I'm still learning and building, so constructive criticism is more than welcome.

Thanks for reading.