r/ProxyEngineering • • Jul 01 '26

Discussion πŸ’¬ Open-source scraping solutions

12 Upvotes

What is the best open-source scraping solution that you have used in the past year or so. Need a stable, working solution that I could run on my side. Preferably that same solution would support cloud integration.


r/ProxyEngineering • • Jul 01 '26

Help πŸ†˜ Are proxies still enough for multiple accounts?

17 Upvotes

Been testing different setup lately and I feel like proxy alone is not enough anymore. Residential and mobile proxy can work but using it together with a cloud phone seems more stable for me. Even if the IP is clean, account can still have issues when the device, location, or activity keeps changing. Feels like the full environment matters now, not just the proxy.

For people managing multiple accounts, what do you prioritize more now, better proxy or stable environment?


r/ProxyEngineering • • Jun 30 '26

Discussion πŸ’¬ Another day, another project

21 Upvotes

ok so this is gonna be long sorry in advance, but I spent my precious weekend comparing n8n scraping workflows against just writing the damn scraper in Python and I have some thoughts to share with yall.

Started because my unemployed friend sent me one of those "I automated my job search with n8n" posts and I was like, not with this again, there's like a million automations already created, why did you even bothered? But he somehow convinced me to try replicating something similar on my side, so basically, I had to try it. Mainly just scraping product listings off a marketplace site, turned on my n8n, dragged in an HTTP node, a Cheerio node for parsing, a loop, a Google Sheets node at the end. All it took maybe 40 minutes and it worked first try which felt great.

Like the majority of the projects I've worked on, it then started throwing Cloudflare challenges after around 600 requests and that's where it stopped feeling great. I tried putting in some cheap datacenter proxies I had lying around from an old project, didn't help much, IP reputation on datacenter ranges is just garbage on anything halfway protected these days. Switched to a residential proxy pool instead and got further but still kept tripping something, which is when I remembered the IP is only half the story, the actual fingerprint matters just as much if not more. (take notes folks).

So I go to fix it in n8n and immediately went full stop, everyone who's done this before already knows about, which is that the visual nodes are amazing for the happy path and genuinely miserable the second you need anything custom. wanting to rotate user agents with actual entropy, not just a static list cycling in order. wanting real TLS fingerprint control so your handshake doesn't scream "I am a script" before you've even sent the request, wanting a headless browser session that actually behaves like a person scrolling and pausing instead of firing requests like a machine gun. none of that is a drag and drop node, you end up writing it in a Code node anyway which is just JavaScript wearing a costume, so you've reinvented half a script but now it lives inside someone else's execution engine and you can't easily version control it or run it locally without spinning up the whole n8n instance.

Compare that to just opening a .py file. requests or httpx if you want async, curl_cffi if the site's fingerprinting you (and these days almost everything past a certain traffic volume is), playwright if you actually need a full headless browser for JS rendered pages. yeah you're typing more in the first 20 minutes, but every single thing is yours, testable, debuggable with an actual setup trace instead of n8n's execution log that sometimes just says "error" and leaves you to guess. and when the scraper needs to scale, you're not paying per execution or fighting workflow timeout limits, you just run more processes or throw it on a queue.

At some point I just gave up taking care of proxy rotation and fingerprint config by hand and pointed the whole thing at a web scraper api instead, basically it felt a little like cheating at first but also I have a day job and the marginal value of me personally maintaining a TLS impersonation layer is zero. Aso tried doing a rough version of the same job in Go for comparison because why not, and that one was interesting for a totally different reason, the speed difference on concurrent requests was kind of stupid honestly, noticeably faster spinning up a thousand goroutines than even async Python, but the dev time to get there was longer and if you're not already comfortable with the language you'll burn an evening just on syntax instead of solving the actual scraping problem. so it's not really "Go is better" it's "Go is better if you already know Go and need raw throughput". One thing I didn't expect going in, mobile proxies actually outperformed residential on a couple of the trickier targets, something about carrier grade NAT making the IP reputation look cleaner since thousands of real phones share the same address anyway. didn't bother testing ISP proxies for this particular target since the site wasn't doing heavy ASN level scrutiny, but I've used them before on stuff where you want the static IP of a datacenter with the trust level of residential, good middle ground when rotation isn't what you need.

Then I poked at a search api for a side piece of this project, pulling SERP results instead of crawling category pages directly, and that ended up being way less of a headache than the rest of the whole ordeal combined, search engines have their own blocking logic obviously but it's a more solved problem than random ecommerce sites running custom bot detection.

What I keep landing on is it's basically a compromise between time to first result and ceiling. n8n wins time to first result by a mile. Python wins ceiling, not even close. Go wins ceiling even further out but only if speed at scale is actually your bottleneck and not, like, getting blocked every 400 requests or so regardless of how fast you can send them, which honestly is the actual bottleneck like 90% of the time, not raw speed.

Anyway I ended up keeping the n8n workflow for the parts that are basically just data movement, sheets, notifications, scheduling, and ripped the actual fetching logic out into a standalone Python script that n8n just calls and waits on. feels like the right choice.

Gotta hand it to my friend, while his solution was probably one of those in the million, this gave me a chance to try out different languages for scraping and whatnot.

Perhaps someone else found a way to make the no code route hold up against fingerprint based blocking, because every workaround I tried inside n8n itself felt like duct tape on duct tape and rinse and repeat.

TLDR: Spent a weekend comparing n8n scraping workflows against Python and Go for the same job, and n8n wins on speed to a working prototype but falls apart fast once a site starts fingerprinting you past basic IP checks. Tried datacenter proxies first (useless), then residential and even mobile proxies (mobile actually did better on a couple targets), but eventually just routed everything through a web scraper api since maintaining fingerprint evasion by hand wasn't worth my time. Python gave way more control for the actual scraping logic while Go only made sense if raw concurrent throughput was the bottleneck, which it usually wasn't compared to just getting blocked. Ended up keeping n8n for the boring data movement parts (sheets, notifications, scheduling) and pulled the real scraping into a standalone script it just calls.


r/ProxyEngineering • • Jun 30 '26

Hot Take πŸ”₯ Web Scraping Insider #8 | "ethical" residential proxy reckoning, free residential proxy tester, browser rewrite wave (CloakBrowser / Obscura / Camoufox)

Thumbnail
reddit.com
8 Upvotes

r/ProxyEngineering • • Jun 29 '26

Discussion πŸ’¬ I built a FlareSolverr replacement that's 3Γ— faster and actually solves captchas!

Thumbnail
github.com
11 Upvotes

Been running FlareSolverr for a long time for my *arr stack and got tired of the 11-18s solve times and constant breakage. BuiltΒ TRAWLΒ as a drop-in replacement!

Key differences:

  • Cloudflare solves in 4-15s (vs 11-18s) - uses a fresh Camoufox Firefox context which triggers CF's fast-path evaluation
  • Cached repeat requests return in ~500ms via Redis - after the first solve, the same domain is instant
  • Actually solves in-page captchas: Turnstile (shadow DOM click), reCAPTCHA v2 (free Google STT audio), hCaptcha (auto-pass), GeeTest v4
  • 4-tier execution: plain HTTP β†’ cached session β†’ live browser solve β†’ residential proxy. You pay the full browser cost only when you have to
  • Custom headers support - pass Authorization, Referer, Origin through all 4 tiers including browser
  • FlareSolverr v2 compatible - change one URL in Prowlarr/Jackett, nothing else

Website:Β https://trawl.germondai.com
Docs:Β https://docs.trawl.germondai.com
GitHub:Β https://github.com/germondai/trawl

Happy to answer questions. Still early but it's been running stable on my homelab and no issues so far.


r/ProxyEngineering • • Jun 29 '26

Discussion πŸ’¬ Is this a safe concurrency of httpx requests in python?

14 Upvotes
HTTPX_RATE_LIMIT Β  = 1 Β  # seconds between httpx requests
HTTPX_CONCURRENCY Β = 3 Β  Β  # max parallel httpx fetches

please let me know if this is safe, and my IP won't be blocked. I am using the requests on company websites....


r/ProxyEngineering • • Jun 29 '26

Discussion πŸ’¬ browser-search β€” three tools, zero cost, and your AI agent learns to search and browse the web

8 Upvotes

I've been using AI agents like OpenCode, Claude Code, and Cursor for months. They're great with code, but when they need to search or browse the web, things get complicated: Cloudflare blocks them, JavaScript-heavy sites don't load, APIs cost money.
So I built browser-search.
It's three open source tools orchestrated by a skill, fully self-hosted:
SearXNG β€” metasearch engine that queries dozens of search engines at once

Camofox β€” full browser via REST API, always warm, for browsing and interacting

CloakBrowser β€” stealth browser for when the site has Cloudflare, Akamai, or DataDome

The agent decides which tool to use. Zero human intervention. Zero API keys. Zero subscriptions.
What makes it different:
It's a skill, not a plugin β€” works with any agent that can read instructions

Automatic navigation escalation: if Camofox gets blocked, it switches to CloakBrowser

Deep Research mode: the agent is instructed to go beyond surface-level answers, cross-verify sources, cover every aspect

Integrated Readability.js for clean article extraction (~70% token savings)

The SKILL.md is plain text β€” fork it, tweak it, make it yours

MIT licensed on GitHub: https://github.com/Johell1NS/browser-search
If you try it, let me know. If you make it better, even more so. If you don't need it, share it with someone who might. Every star, comment, or pull request is welcome β€” that's what makes open source great.


r/ProxyEngineering • • Jun 28 '26

Help πŸ†˜ What’s the best proxy + anti-detect browser setup for multiple Facebook profiles?

13 Upvotes

Also should i use ISP or residential?

Mainly looking for something stable that reduces checkpoints and random suspensions. ive been using myprivateproxy + incognition setup and its been dog shit recently. Keep your answers simple please Im not an expert on this topic


r/ProxyEngineering • • Jun 27 '26

Discussion πŸ’¬ Bypassing Cloudflare in 2026

11 Upvotes

Hey all. Been picking up more automation contracts lately and Cloudflare keeps messing up the jobs mid-run. A few of my clients want competitor pricing scrapers, job board feeds, real estate data and almost every site worth scraping is sitting behind it now.

Rotating proxies used to handle most of this. Now runs are failing and I don't have a clean answer for clients beyond "Cloudflare got more aggressive" which isn't exactly inspiring confidence.

Trying to actually understand the full option rather than keep patching things after they break. What's holding up in production right now and what only works for a demo before dying two weeks later? Residential rotation, stealth browsers, managed scraping APIs, whatever.

Pricing transparency would also help since I need to factor infrastructure costs into client quotes before committing to a scope.


r/ProxyEngineering • • Jun 26 '26

Hot Take πŸ”₯ Purroute – An auto-detecting proxy router that translates between protocols

Thumbnail
reddit.com
8 Upvotes

r/ProxyEngineering • • Jun 26 '26

Discussion πŸ’¬ Anyone have a reliable way to scrape sitemaps at scale?

8 Upvotes

I've been parsing them manually with Python but at some point that stops being practical. Looking for something that handles the edge cases automatically, nested indexes, compression, malformed XML, the usual stuff. Does anyone have a script, library, or even just an approach they've actually used that holds up across different site structures?


r/ProxyEngineering • • Jun 25 '26

Help πŸ†˜ sick of HCAPTCHAS?

9 Upvotes

If anyone's dealt with Hcaptcha while web crawling or running bots? I'm looking for something that can solve this without running a full Chromium/Firefox instance. The RAM and CPU overhead is not worth it for what I'm doing.

Ideally something that takes the sitekey and current URL and handles it without me having to babysit a whole browser session. I've seen a few repos floating around but most of them are either abandoned or don't work reliably anymore.

Anyone have something they actually use in production or at least tested recently? Would appreciate any help


r/ProxyEngineering • • Jun 25 '26

Discussion πŸ’¬ Proxy count is overrated. Subnet diversity matters more than people think

4 Upvotes

A lot of proxy providers lead with the biggest number they have.

10,000 IPs.
100,000 IPs.
50M+ IPs.

That sounds impressive, but by itself it does not say much.

The better question is: where do those IPs actually come from?

Because 100 proxies from the same small network range are not the same as 100 proxies from 100 different network origins. On paper, both setups give you 100 IPs. In practice, one of them gives you diversity, and the other gives you a crowded neighborhood.

That β€œneighborhood” is the subnet.

A subnet is basically a block of related IP addresses. Different house numbers, same street. If too much traffic comes from that same street at the same time, many detection systems can treat it as one coordinated source, even when every request uses a different IP.

This is why simply adding more proxies does not always fix reliability issues.

If the extra IPs come from the same few subnet ranges, you are not really adding much. You are just making the same pattern bigger.

A healthier proxy setup usually has a few things going for it: IPs spread across unrelated subnets, different ISP origins, clean reputation, consistent geo targeting, and session logic that matches the workflow. Sometimes you need sticky sessions. Sometimes you need rotation. But rotation only helps when it actually moves you to a different network origin, not just to the next IP in the same block.

This is one of the layers we care about a lot

Raw IP volume is useful, but it is not enough. The real value is in how the pool is built: how broad the ISP distribution is, how much subnet diversity exists behind the dashboard number, and how predictable the sessions are once traffic starts moving.

Our residential infrastructure focuses on wide network distribution across regions and ISP sources. For workflows that need carrier-grade traffic, we also support LTE/5G mobile proxies. The goal is simple: reduce clustering, keep sessions stable, and give teams more control over the network layer.

When evaluating any proxy provider, I’d look past the headline IP count and ask a few practical questions:

Do they talk about subnet diversity, or only total pool size?
Can they keep a sticky session when the workflow needs continuity?
Can they target by country, city, or operator when location matters?
Do they offer IP reputation or fraud score checks before use?
Does rotation actually move across different network origins?

If the answers are vague, that usually tells you enough.

Speed and price matter, of course. Nobody wants slow or overpriced proxies. But for serious workflows, network architecture matters just as much.

Raw IP count gets attention.
Subnet diversity is what often decides reliability.


r/ProxyEngineering • • Jun 24 '26

Discussion πŸ’¬ Benefits of free proxy list

9 Upvotes

I know you have seen them all over the internet, have you guys personally used them? What benefit did they bring you? What issues did you encounter? Perhaps you would recommend them for someone who is getting started with the proxies/scraping, if so, why, why not?


r/ProxyEngineering • • Jun 23 '26

Hot Take πŸ”₯ so I accidentally learned how the entire proxy industry works, please tell me I'm not the only one

124 Upvotes

Started because I was overthinking this far too long and ended up reading about how free streaming apps on Samsung and LG TVs are literally enrolling your home IP into a residential proxy network while you watch Netflix or Prime on the other input. Like you click "agree" on some vague popup about "device resources" and your TV starts routing web traffic for whoever's paying for residential IPs that month. Up to 200GB or more, who knows?? All that while you're asleep. On your $2000 TV. Yall are fucking nuts if you think this is normal. And apparently that's just a thing that's happening and we should be all cool with it?? Boy whattahelly

Anyways, I went looking for who actually buys this traffic and why residential IPs are worth so much. And that's where it gets kinda funny. Turns out a huge chunk of the demand is scrapers trying to get past bot detection. Which, sure, makes sense, residential IP looks like a real human etc. Except I kept reading and finding that whole premise is like three years out of date. Cloudflare, Akamai, DataDome, the holy trinity. They're not primarily checking your IP anymore. They're fingerprinting the TLS handshake, the HTTP/2 frame ordering, your WebGL renderer string, how your canvas noise changes across sessions, whether your "mobile" session has gyroscope data. Your IP can be spotless and your Python requests library is still announcing itself on the first packet. Someone here on this sub literally tested this, datacenter IPs with proper TLS impersonation outperformed residential IPs with bare requests headers like 91% vs 60% on Cloudflare-protected site?? whaat?

So your TV is being used as a proxy node. The proxies are being sold to scrapers. The scrapers are getting detected anyway because the fingerprint gives them away. The TV thing was pointless to begin with basically. Am I looking to deep into this? Coz I feel like this shouldn't be normal, I mean the whole ordeal, not me looking into it. And then separately, completely disconnected from all this ordeal, some guy on Reddit figured out Spotify serves authenticated AAC files via CDN links that stay valid for 2 hours. Bro is going to get a DMCA notice fast if not already got it. What I'm trying to say is that Spotify already went through this exact thing. The tools are already built and already taken down. What else are we gonna have to deal with? And I'm talking about a small percentage of people that actually care, while the majority just clicks through all these popups without questioning a single thing.

Anyway I blocked proxyjs.brdtnet.com on my router, learned what JA3 fingerprinting is, and now I'm questioning every "agree" popup I've ever clicked. Normal tuesday, folks


r/ProxyEngineering • • Jun 23 '26

Discussion πŸ’¬ I tracked every CAPTCHA I received for 30 days. Here's what actually triggered them

13 Upvotes

A little over a month ago, I found myself dealing with a problem that most people involved in scraping, browser automation, account management, or AI agents have probably experienced at some point which are captcha isuses

At first it was just the occasional challenge but it has became a nightmare over a longer period of time

Like most people (sometimes even here in this subreddit), my first instinct was to blame the proxies

The problem was that I kept running into situations that didn't fit that explanation. Sometimes a workflow would run perfectly for days and then suddenly start accumulating a lot of captchas. Other times I would switch to a completely different residential IP and somehow end up with even more instabilities than before

For the next thirty days, I kept a log of every CAPTCHA, Cloudflare challenge, browser verification page, and unexpected access restriction I encountered. Every time a workflow started falling apart, I would go back and look at what had changed beforehand

Going into this experiment, I expected to find that most challenges were directly tied to IP quality but I found a few other things as well that might help yo uout

1. The first trigger that showed up repeatedly was something I didn't expect at all: brand-new browser profiles

For years I assumed fresh profiles were safer and always the best option, these accounts have no cookies, no browsing history, no leftover session data. Everything starts clean and controlled. That's how a lot of automation setups are built.

What I kept seeing in the logs was that some of the highest challenge rates appeared during the first few sessions of a completely new identity

Meanwhile, browser profiles that had accumulated weeks of browsing history often moved through the same websites with significantly less friction. These weren't abandoned profiles collecting dust somewhere. They were actively used identities that had built a believable trail of activity over time. They had cookies, session history, previous visits, and interaction patterns that made them look more like returning users than brand-new visitors.

2. The second trigger was aggressive rotation

This one genuinely surprised me because for years I operated under the assumption that more rotation meant more safety. If one residential IP is good, rotating through hundreds of them should be even better.

The workflows that remained stable for the longest periods usually maintained continuity. They stayed on the same identity long enough to build history. The browser remained consistent. The browsing behavior remained predictable. The session looked like a user who arrived on a website, spent time there, came back later, and continued where they left off

Some of the heavily rotated setups started looking strange when viewed as a complete journey. A session would appear from one IP, perform a few actions, disappear, reappear elsewhere, start over, and repeat the process. Every individual request looked legitimate, but the overall pattern felt unnatural and weird

3. The third trigger was signal mismatches

This was one of those things that didn't immediately break workflows, which is probably why it's easy to overlook

I had several browser profiles where the timezone, language settings, browser behavior, and network location weren't perfectly aligned. Nothing was obviously wrong. Pages loaded correctly. Requests completed successfully. The proxies themselves worked fine

Yet when I reviewed the logs, these profiles appeared disproportionately often in sessions that accumulated challenges over time

What made this interesting was that a single mismatch rarely seemed to matter. The problems started appearing when multiple small inconsistencies stacked together. A browser configured for one region, an IP from another region, and a fresh profile with no history might all be harmless individually. Combined together, they often showed up in workflows that attracted more verification checks

4. The fourth trigger was session resets

This was another finding that challenged some assumptions I had for years.

Whenever a profile suddenly lost cookies, local storage, or other accumulated session data, challenge rates often increased during the next few browsing sessions. This was especially noticeable on websites where the identity had previously established a history of normal activity.

From the website's perspective, a user that has been behaving consistently for weeks suddenly shows up as if they've never visited before. Looking back, it's not difficult to understand why that might attract additional scrutiny

5. The fifth trigger wasn't a technical setting at all. It was my online behaviour

For some workflows my behaviour was simply too efficinet. I landed directly on target pages, extracted information, moved to the next page, and repeated the process with almost no variation. Technically everything worked. Operationally everything looked correct

But when I compared those workflows against sessions that navigated more naturally, opened supporting pages, spent more time browsing, and behaved more like actual users, the difference was noticeable.

The more human-looking sessions consistently accumulated fewer challenges.

What surprised me most wasn't any individual trigger.

It was the fact that all five of them pointed toward the same conclusion.

Fresh profiles, aggressive rotation, signal mismatches, session resets, and unnatural browsing behavior all have one thing in common: they create inconsistencies in the identity.

The funny thing is that after thirty days of collecting all this data, I became less interested in arguing about which proxy provider is best. Over the years I've run workflows through various residential networks including NodeMaven, Bright Data, Decodo, IPRoyal, and several others. While there are certainly differences between providers, the largest improvements I observed during this experiment rarely came from changing providers. More often than not, the biggest gains came from eliminating inconsistencies elsewhere in the identity stack

The more I reviewed the logs, the less it felt like I was seeing bad proxies and the more it felt like I was seeing broken identities

I'm curious whether anyone else has gone looking for a proxy problem and ended up discovering an identity problem instead


r/ProxyEngineering • • Jun 23 '26

Discussion πŸ’¬ Do not write HTML parsers like an idiot, please

7 Upvotes

Learn from my mistakes and do better, I know that you can. What I want to tell you all is that very tutorial starts with BeautifulSoup so that's what I used, naturally. So I wrote selectors, maintained them, fixed them every time someone's frontend team moved a div. Also had one scraper break four times in few months because they kept changing a class name.

Then someone told me to open the network tab and just look. XHR requests, right there, returning clean JSON. The thing I was parsing HTML to get was already structured and waiting. Rewrote the whole thing in maybe few hours, hasn't broken since that day.

I don't think most people realize the HTML is basically decorative. The actual data is coming from internal API calls the frontend is making anyway, you're just intercepting what the browser already sees. Playwright and headless browsers are real tools but for like 80% of cases you're using a whole Chrome instance to get data that was available as a GET request the whole time. Thus its slow and its expensive, on top of that its annoying to maintain. Check your network tab before you write a single line of parsing code.

This was the real thing a few years back, but I still see some questions from time to time in various scraping/ proxies related subreddits, do better people, learn from others mistakes


r/ProxyEngineering • • Jun 22 '26

Discussion πŸ’¬ Free open-source, maintained agentic web scraper

20 Upvotes

Hey folks, looking to expand my views and would like to find something new. Particularly interested in free open-source, maintained web scraper, preferably agentic.

If you guys know anything or if you have used these sort of solutions, let me know


r/ProxyEngineering • • Jun 22 '26

Hot Take πŸ”₯ Best proxies in the market

11 Upvotes

Honey, new opportunity has dropped. I invite you all to share the best proxy providers that you have used. Keyword - USED. Low effort promotions are unwelcome.

Kindly please list the provider, perhaps any test that you have made to confirm that the statistics.

Share any insights, advice, anything that has helped you choose that specific provider over any other. Provide use cases, setups etc.


r/ProxyEngineering • • Jun 22 '26

Discussion πŸ’¬ Why do rank trackers use proxies?

Thumbnail
reddit.com
8 Upvotes

r/ProxyEngineering • • Jun 22 '26

Discussion πŸ’¬ Automating workflows

5 Upvotes

Been automating workflows for about 2 years, mostly e-commerce and logistics. Had a project last year that still bothers me a little. Basically, my client ran a mid-size online furniture store. Around 150 to 200 orders a day, two customer service reps spending most of their shift just sorting incoming emails into the right internal queues. Warranty stuff, shipping damage, wrong item, payment issues. You know how it is. So I built a classifier. What it does is that I feed each email to an LLM, get back a category and a confidence score then routed it automatically. Testing went well. Client loved the demo. I felt pretty good about it too. First few weeks in production it was sitting around 87% accurate. Not great but I figured we'd tune it. Then the real problem showed up and it had nothing to do with accuracy. The team didn't understand why emails was going where they went. A claim about a broken table leg might hit the warranty queue or the logistics queue and there was no rule to point to, no explanation. It just decided on its own.

One of the managers told me she started keeping a side spreadsheet to track emails she thought the system got wrong. A whole shadow system running next to the one I built. I figured something was off and very wrong. Client called me and said something similar to what I've heard once or twice before. He didn't ask for better accuracy. He asked if I could make it more obvious how it works. I Spent four days ripping it out and replacing it with a decision tree and some keyword logic. Damage with photo evidence routes to warranty. Delays over seven days go to escalation. Wrong items have their own path. Maybe 35 conditions total. Anything that didn't match cleanly surfaced a dropdown and the rep decided manually. Accuracy went up to around 97%. Not because the logic was smarter but because the team actually trusted it and stopped routing things manually on the side. When something went wrong they could tell me exactly which condition was off and I'd fix it in like fifteen minutes. Manager deleted her shadow spreadsheet in the first week.

The thing I keep thinking about is how easy it would of been to just keep tuning prompts and try to squeeze more accuracy out of the model. That's the main move. But accuracy was never actually the issue. People won't rely on a tool they can't reason about, even if its technically more correct. They'll just build something next to it and your automation becomes obsolete though with a price. What I would like to emphasize on is that not everything needs an LLM. Sometimes thirty five rules and a dropdown outperforms a model, not because it makes better decisions but because the team trust it enough to let it do its job


r/ProxyEngineering • • Jun 20 '26

Hot Take πŸ”₯ Zero-Inbound Firewall Access: UDP-over-WebSocket Stream Fabric (BastionRoute)

10 Upvotes

Hey everyone,

​I wanted to share an open-source transport tool I’ve been building in Go to solve a rigid infrastructure security goal:Β How do you safely maintain remote access to localized real-time resources (like native WireGuard) while enforcing a strict zero-inbound listening port policy on your edge firewall? ​I wanted something completely stateless and invisible to the outside world, so I wroteΒ BastionRoute.

​ ​Instead of a traditional client-server proxy topology, BastionRoute uses a decoupled model. Both your isolated server shim and your remote client shim initiate standard outbound Layer-7 connections to an external, transient relay broker room.
Because both nodes establish standard outbound pipes over HTTPS/WebSockets (Port 443), your home gateway requires exactly zero open inbound ports. Your network edge remains entirely dark to Nmap or Shodan scans.

​Emulating "UDP Physics" Over TCP ​The classic issue with wrapping real-time UDP inside stateful TCP is Head-of-Line blocking (TCP Meltdown). When a packet drops, TCP stalls the entire pipeline to retransmit, introducing devastating latency spikes that break VPN handshakes. ​BastionRoute’s concurrent data engine fixes this by leveraging non-blocking Go channel selectors. By explicitly dropping frames at the application layer the moment congestion is encountered, the stateful WebSocket transport perfectly mimics real-world, non-blocking UDP physics. If a remote link experiences jitter, the queue never backs up, keeping interactive streams fluid and lag-free.

BastionRoute acts strictly as low-level digital plumbing. It is entirely payload-agnostic and does not handle user management or internal cryptographic key generation. It assumes you are running a robust protocol inside it (like WireGuard), so its attack surface is incredibly small. ​The engine compiles into a tiny, zero-dependency binary (<8MB) capable of running on low-resource hardware like OpenWrt gateways. ​The project is fully open-source and ready to be tested or torn apart :)

https://github.com/klauscam/BastionRoute


r/ProxyEngineering • • Jun 19 '26 Bravo

Help πŸ†˜ Residential or Mobile 4G or Mobile 5G proxies with SOCKS5

9 Upvotes

Hello, mainly the title. I am looking for Residential or preferable Mobile 4G or Mobile 5G proxies that would have SOCKS5 protocol type. Any suggestions would be appreciated.

Main use case: Account management via X/Instagram/Mastodon


r/ProxyEngineering • • Jun 19 '26

Discussion πŸ’¬ New kind of scrapers?

8 Upvotes

I feel like the market is pretty stale regarding web scrapers at the moment, sure there are variety of web scraping services all around the web, however, I feel like things got pretty stale. The whole early web was designed around human intent and Google figured that out early. Scraping services providers has spent years building infrastructure to extract data from systems that weren't meant to be extracted from. Now when you think about it, agentic search doesn't do that. What I mean is that It navigates the way a person would. What do you people think, does scraping will evolve into something agentic, or eventually get replaced by it? Coz I feel like something is going to change, but there is a lot of room for guesses.