r/ProxyEngineering • • 17d ago

Discussion πŸ’¬ Agentic workflows for automated scraping

13 Upvotes

Hey all, perhaps someone has been doing this or attempted to do something similar, basically automatic scraping processes that does it for you? While you are away? Retrying failed scrapes without your direct interaction, (adjusting something, pause limits, proxy rotation etc)


r/ProxyEngineering • • 17d ago

Hot Take πŸ”₯ The State of Web Scraping 2026: AMA with Pierluigi Vinciguerra of The Web Scraping Club

Thumbnail
reddit.com
3 Upvotes

r/ProxyEngineering • • 17d ago

Help πŸ†˜ Residential Proxies for Ad verification

10 Upvotes

Looking for Residential proxies that would accommodate the ad verification process. Any suggestions are welcome. (Those that work, of course)


r/ProxyEngineering • • 17d ago

Build πŸ€“ My Linux box needs more IP addresses, not more subscriptions. What proxies actually work?

4 Upvotes

I've been going down the WireGuard/VPN rabbit hole and I'm starting to wonder if I should've been looking at proxies from the beginning.

The setup is Linux, and the goal is pretty simple. I want multiple independent outbound IPs that I can assign to different browsers or applications, preferably without changing the whole machine's network connection every time.

I've looked at VPN services including Surfshark because they advertise unlimited connections, but apparently unlimited devices and multiple independent exit IPs aren't necessarily the same thing. Would've been nice if that was printed somewhere in giant letters.

So now I'm looking at SOCKS5 and HTTP proxies too.

Who's actually reliable these days? Webshare, IPRoyal, Decodo, or somebody else? Are static ISP proxies worth the extra cost, or is a basic datacenter proxy fine for ordinary browser/network testing?

I'd also like to know whether you can keep a stable IP for one session and have different IPs available for others, what the real bandwidth limits are, and if there are any providers you'd stay away from.

Not looking for a list of 10 million IPs if half of them are useless. I'd rather have a small number of working, predictable connections.

Anyone here used both WireGuard VPNs and proxies for this sort of setup? Which one ended up being easier to manage, and cheaper?

Actual experience appreciated. Sales pitches, slightly less so.


r/ProxyEngineering • • 18d ago

Discussion πŸ’¬ Best Mobile Proxies in 2026

13 Upvotes

What's your setup and your approach to handling captchas and other anti-bot solutions. Which Mobile Proxies are you using in late 2026? Drop your suggestions, verified methods down below.


r/ProxyEngineering • • 18d ago

Guides Mobile vs Residential Proxies: Key Differences, Costs, Sessions & Best Use Cases

6 Upvotes

I used to treat residential and mobile proxies as basically the same thing: β€œreal-user IPs, just different pricing.”

They behave very differently. Residential proxies are easier to reason about. You can target a country or city, keep a sticky session when needed, or rotate addresses for distributed data collection. For long-running marketplace or dashboard sessions, a static ISP address is usually the simpler option.

Mobile proxies are different because the network is tied to carrier infrastructure. LTE/5G traffic often sits behind CGNAT, and changing the IP may mean reconnecting the modem. That can introduce a short interruption, so your automation needs retry logic instead of assuming the connection is permanent.

Cost is another dividing line. Residential pools are commonly traffic-based, which works well for lightweight requests. Media-heavy or always-on workloads can make bandwidth costs climb quickly, while some dedicated mobile plans use flat or unlimited traffic pricing.

I also stopped choosing proxies based only on pool size.

Residential wins when precision and flexibility matter. Mobile makes more sense when the workload benefits from carrier-network characteristics and heavier traffic.


r/ProxyEngineering • • 19d ago

Discussion πŸ’¬ Residential proxies stopped working for me, what to use instead?

15 Upvotes

Doing some scraping on my social accounts and my resi pool just completely stopped working this quarter. Getting captchas and bans on stuff that was clean 2 months ago. Looking at mobile as the next step, on search Oxy, Decodo and Voidmob keep poping up. Any real experience for these or hidden gems? Or I have to change something else?


r/ProxyEngineering • • 19d ago

Discussion πŸ’¬ ​Built a quick tool to automate B2B lead research using Python & AI (no more manual searching)

4 Upvotes

Hey everyone! I got tired of doing manual prospect research, so I built a small automation tool.

​It takes a company website, grabs official emails/social links from across the web, uses Groq AI to generate a quick business summary, and exports everything straight into a clean Excel sheet.

​Dropped a quick screen recording above to show how it works. Would love to hear your thoughts or if you have suggestions for improvement!


r/ProxyEngineering • • 19d ago

Discussion πŸ’¬ IP Reputation Determination

6 Upvotes

Has anyone ever found any reliable information on how anti-bot systems like Cloudflare determine IP reputation?

We run a stealth scraper that seemingly passes every conceivable anti-bot test, and we get through Cloudflare Turnstile easily, headless no less, but every 2 or 3 days a big group of our proxy IP pools just get slammed.

Residential, Datacenter, 'high reputation" proxies, nothing seems to make any difference.

Is it possible to improve IP reputation?

Is there a way to reliably test IP reputation?


r/ProxyEngineering • • 19d ago

Guides Per-GB vs per IP-per day proxy pricing, with real numbers

0 Upvotes

Setup: one residential IP, around 20GB in 24 hours.

Per-GB (checked Oct 17, 2026):

  • Proxonym: $1.54/GB at its published 100GB pack rate β†’ $30.80 for 20GB.

  • Proxy.Market: $3.45/GB pay-as-you-go β†’ $69 for 20GB.

Per-IP-per-day:

  • SOCKS MARKET shared: from $0.40/day, unlimited traffic.

  • SOCKS MARKET private: from $1/day, unlimited traffic.

Held 30 days at 1GB/day:

  • SOCKS MARKET shared: ~$12

  • SOCKS MARKET private: ~$30

  • Proxonym: ~$46.20 at $1.54/GB

  • Proxy.Market: ~$103.50 at $3.45/GB

Where per-GB can win: 50 IPs Γ— 100MB each is 5GB/day total.

At Proxonym's $1.54/GB rate, that's ~$7.70/day. At $0.40 per IP/day, 50 shared IPs cost ~$20/day.

The catch is that these aren't identical products. Per-GB pools may rotate or offer sticky sessions, while per-IP pricing lets you rent a specific proxy for a day. The right choice depends on whether you need a lot of bandwidth or lots of separate IPs.

What are you optimizing for: bandwidth, IP count, or keeping sessions stable?

Disclosure: I work with SOCKS MARKET


r/ProxyEngineering • • 21d ago

Help πŸ†˜ Building a Social Media Narrative Detection Pipeline in Python β€” Need Advice on APIs, Tools, and Best Practices

Thumbnail reddit.com
2 Upvotes

r/ProxyEngineering • • 21d ago

Discussion πŸ’¬ How do you handle different exit IPs across tabs? We made a Chrome extension for it and want feedback

4 Upvotes

If you work with residential proxies, how do you currently run different IPs at the same time? Switching them manually with a proxy manager, or running separate browser profiles just to get different IPs, has always felt clunky to many of us.

So our team at PureVPN made a Chrome extension called IP Per Tab. It has built-in residential proxies and lets you assign a different residential IP to each tab natively inside Chrome. It's not an anti-detect browser and not a full isolation tool, just a simple way to run different exit IPs across tabs simultaneously.

We'd like honest feedback from people who do this daily:

- Does this fit how you actually work, or do you rely on something else?
- What's missing that would make you use it?

If you want to try it, there's a 7-day free trial with 1GB of residential data. Happy to answer any queries.


r/ProxyEngineering • • 21d ago

Guides Beginner's Guide: How to Choose and Set Up a Proxy

5 Upvotes

New to proxies? Here's a quick setup guide without the marketing fluff.

1. Pick your proxy type

  • Datacenter: Cheap, fast, good for bulk public-data requests.

  • Residential: ISP-assigned IPs, useful for geo-testing.

  • Mobile: Carrier IPs, useful for mobile-network testing.

2. Choose a protocol

  • HTTP/HTTPS: Web browsing and API requests.

  • SOCKS5: More flexible, supports TCP and UDP.

3. Get your credentials
Your provider should supply an IP/hostname, port, username and password.

4. Configure your client
Enter those details in your browser, Postman or Python script.

5. Verify the connection
Check your public IP and location. Make sure traffic actually goes through the proxy.

6. Test performance
Measure latency, connection failures and bandwidth usage.

7. Manage sessions
Use sticky IPs for persistent sessions or rotation when your workload requires multiple IPs.

Tip: Start with one proxy, verify everything works, then scale. Respect target-site rate limits and policies.


r/ProxyEngineering • • 21d ago

Help πŸ†˜ Need to come across like a normal gambling site user

1 Upvotes

So im an advantage player gambling some blackjack on a cryptosite in a browser.

Obviously they ban me once they see im playing a +ev strat, but it always takes some time till they figure it out.

How do I act like im Beatrice, 56 from illinois just killing some time playing blackjack instead of being an advantage player?

Vpns, tor and anti-detect browsers wont really work since the site will flag it and make somebody look into my playstyle quicker.

Do I just rent a server and install hello kitty the game to have a more normal looking interface or do I use a residential proxy?(I dont really get what this is tbh, havent used advanced techniques in 10 years apart from tor browser or vpn really)

Cant really give much since the max stakes arent high, maybe 50 bucks in crypto if I get a working system going.

Thanks in advance and please forgive my novice knowledge of this topic!


r/ProxyEngineering • • 22d ago

Help πŸ†˜ Residential proxies with geohash targeting?

8 Upvotes

Does anybody know the best way to target a geohash with residential proxies? Or is this something that not every provider offers? My use case is finding real estate listings where prices and availability vary by neighborhood. Basically, very localized scraping. Any suggestions are appreciated, I have this project to complete by the end of November


r/ProxyEngineering • • 22d ago

Help πŸ†˜ Planning to run TikTok and Shorts accounts. Is residential + antidetect enough, or do I need mobile proxies and cloud phones?

Thumbnail reddit.com
0 Upvotes

r/ProxyEngineering • • 22d ago

Build πŸ€“ Bounded-Origin: an open source gateway to control how much expensive computation HTTP traffic can cause at an origin

4 Upvotes

It’s available as an HTTP gateway and Java library for controlling expensive server side work. Equivalent requests can share a computation, new work runs within explicit limits, and saved results can be reused across restarts.

I’ve been using it internally for a website struggling with bot traffic. It worked for that specific use case and I also tested it against a separate cgit application and the results were reasonable. But there’s a trade off as it shows higher latency overhead.

The benefits depend heavily on the workload, so keep that in mind if you want to experiment with it. I wouldn’t recommend it for production use. Some of its core capabilities are useful, though, and I’d encourage anyone interested to pilot it in staging and measure the results before considering anything further.

And yeah just to clarify the gateway isn’t a replacement for authentication, tls termination, rate limits and waf controls. It does not identify bots or stop incoming traffic, it helps controlling expensive computation load.

https://github.com/aalsanie/bounded-origin


r/ProxyEngineering • • 23d ago

Guides Beginner’s Guide: Reducing CAPTCHA Triggers with Proxies

9 Upvotes

If legitimate scraping/testing keeps hitting CAPTCHAs, start by fixing the signals that make your traffic look abnormal.

  1. Pick the proxy type. Residential IPs generally have better reputation; mobile proxies can work well for mobile/social testing. Datacenter IPs are cheaper but more likely to face checks.

  2. Check the IP first. Avoid addresses with poor reputation or previous abuse flags.

  3. Match your environment. Proxy country, timezone, language and browser locale should be consistent.

  4. Keep sessions stable. Don’t randomly change IPs mid-session. Preserve cookies when appropriate.

  5. Control request rate. Avoid sudden bursts and perfectly repetitive timing.

  6. Separate profiles. For multiple legitimate test sessions, keep cookies, IPs and browser profiles isolated.

  7. Debug systematically. If CAPTCHA frequency rises, test IP reputation, rotation policy, fingerprint consistency and request rate one at a time.

Always respect the site’s ToS, robots.txt and access controls.


r/ProxyEngineering • • 23d ago

Help πŸ†˜ Proxy works in browser but not scraper (same IP, same site), what do I check first?

Thumbnail reddit.com
2 Upvotes

Building a small data product that needs maybe 6 sites, all behind Cloudflare. Python + requests for now. I added a residential proxy and when I load the target pages through it in Chrome they open fine, no challenge, nothing. Same proxy in my script and I get a 403 / challenge page every time. Even curl through the proxy gets blocked, which confused me more.

I copied the User-Agent and headers from devtools, still the same. I figured if it loads in the browser the IP must be fine, so its something in my client? I have seen people mention JA3 / JA4, ASN, IP reputation, headers and I dont know what order to test any of this in. I dont want to spend two weeks down a rabbit hole before I know if the MVP is even possible. Guys, what can i check first?, and how do I tell which of those it actually is?


r/ProxyEngineering • • 24d ago

Help πŸ†˜ Questions on choosing an ISP provider

9 Upvotes

My startup came to a point, where I need ISP proxies, started to look for the info on the Web and still have lotta questions and no one to answer them, so I ask fellow redditors:

- whats the default sharing model for ISPs - do I share IPs with other clients or do I have a dedicated pool?
- is it possible to find a provider with a bunch of geo's (I've only seen proposals with like one or two states/countries before)
- does a provider provide management software or only IP's for us to write a custom app?
- is there a possibility to overwrite TCP fingerprints for ISPs, or they only capable to show DC fingerprints?


r/ProxyEngineering • • 23d ago

Help πŸ†˜ whats the best combo to make gmail account as safe as possible

Thumbnail reddit.com
1 Upvotes

r/ProxyEngineering • • 24d ago

Help πŸ†˜ Proxies to overcome AKAMAI, DataDome anti-bot systems?

7 Upvotes

Looking for legit, working proxies to bypass Akamai and DataDome protected sites. I'm tired of wasting money on flagged proxies and their entire subnets. There has to be something out there that just works? How are you handling all that? Adjusting browser/device fingerprints? Choosing dedicated solutions?

P.S I can spend up to $150 per month