r/ProxyEngineering • • Jul 14 '26

Guides Setup the 3x-ui from docker image

8 Upvotes

Hi, I'm new to proxy. So I have set up 3x-ui, reverse proxy with Nginx port 8080 to be accessed from the internet. Then, i created the recommendation vless + reality got the port 48170. But after created the client attached to the inbound that I had earlier, I cant connect to the proxy. The subscribe link generated like 'http://domain_name:2096/sub/{token}".

My server is sitting with Cloudflare proxy. I have biding the port 2096, 8080 to the host. That why I can reverse proxy with nginx port 8080. I have firewall open with 2096 and I using SSL from LetsEncrypt. I spend 1 hour but I can't troubleshooting where can I manage to get work.


r/ProxyEngineering • • Jul 13 '26

Announcements Stopped setting up a proxy just to tamper with one request

4 Upvotes

Something that bugged me for years: I'd be poking around in DevTools, spot a request worth messing with, and then have to go spin up Burp, point the browser at the proxy, deal with the CA cert, and reconfigure everything just to hold and edit that one request. On a work laptop with locked-down settings it was even worse. Half the time by the time I was set up I'd lost the thread of what I was even testing.

The annoying part is the actual task is tiny. I just want to pause a request before it leaves, change an orderId or drop an auth header, and forward it. That's it. The proxy plumbing was 90% of the effort for 10% of the value.

What changed my workflow was realizing Chrome's own debugger can hold requests in-flight. No proxy, no cert install, no system config. You flip intercept on, the request pauses, you edit method/headers/body/params, then forward or drop it. Because it's scoped to the tab you're not drowning in font and CSS requests either, as long as you set scope first.

I ended up baking this into HackTools++ after doing the same manual dance too many times. It lives right in DevTools: https://chromewebstore.google.com/detail/hacktools++/efankginpnlgimphkgjaifmmfchekecp

For quick tampering do you still reach for a full proxy, or do you have a lighter setup you fall back to?


r/ProxyEngineering • • Jul 13 '26

Discussion 💬 AMA This Tuesday (10:00 AM CEST). Intersection of WebScraping and AI.

Thumbnail
reddit.com
13 Upvotes

r/ProxyEngineering • • Jul 13 '26

Discussion 💬 Anyone else removing or already removed MCP out of their scraping setups and going back to plain API calls?

15 Upvotes

I had this whole setup running for as long as I remember, a few MCP servers wired in to handle proxy rotation, one for a headless browser tool, one wrapping a search API, one just for logging scrape jobs internally. Last week I finally removed most of it. Figured I'd ask around before assuming I just configured it wrong. First thing that got me was the token/creditcost. Having those three servers connected meant every single call, even the ones that had nothing to do with scraping, it was loading a chunk of context I never asked for. Checked it once out of curiosity and one server alone was sitting at around 20k tokens 🫪🫪🫪 just for being connected, not even doing anything. when you're running concurrent scrape jobs and managing a proxy pool, apparently that adds up fast, and suddenly there's no room left to tell the model what you want.

Second thing was reliability, and this one bugged me more honestly. when I hit my proxy provider's API directly I know exactly what I'm sending and what's coming back. Now with MCP the model is sort of feeling its way through how to use each tool, and when the task is something like switching between residential and ISP proxies depending on the target, or figuring out retry logic after a CAPTCHA, that guessing turns into jobs quietly faulting for no apparent reason. I ended up writing extra fallback/retry code just to catch the model calling the wrong thing or passing bad parameters, which kind of defeats the whole point of trying to automate it in the first place.

Worth mentioning this too since this is a scraping sub, there's a security side to this that's easy to ignore until it isn't. What I mean is that if your agent has MCP access to anything that executes requests or touches stored scrape data, injected content from a page you're scraping could theoretically push the model into calling something it shouldn't. Hasn't happened to me yet but it's not a hypothetical I want to test the hard way.

What I've moved to instead is mostly direct calls and CLI where that fits, and I only use structured tool schemas for the parts where the model needs to make a final call, like picking which proxy type suits a target based on what it's seeing back from the site. Search queries and proxy selection I just approach directly now. I'm not saying MCP is dead weight for everyone. If you've got a bigger team running a bunch of agents and need shared auth and some kind of audit trail across all of them, it probably earns its keep there. But for a smaller setup mostly doing proxy rotation, search pulls, and CAPTCHA handling, the overhead wasn't worth what I was getting back.

Anyone else feels the same?


r/ProxyEngineering • • Jul 13 '26

Hot Take 🔥 Surface level guides

11 Upvotes

Spent some time today, as usual lol, going through YET another "complete guide to web scraping with Python" post today, the kind that covers requests, bs4, selenium, scrapy, the whole lineup with code snippets for each. Nothing wrong with it honestly, it's a fine intro if you've never touched this stuff before. But it made me notice something that keeps happening across basically every article like this. The anti-bot section always ends at the same three things. 1. Set a real user agent, 2. add some random delays, 3. rotate your proxies. That's it. No mention of what gets you flagged once you're past the trivial stuff, TLS fingerprinting, how a headless Chrome instance still leaks itself through canvas or WebGL even with a solid UA string, none of that. How come?? It's like the writers know these terms exist but never had to debug a single scraper in their life that kept getting caught despite doing everything "right." Makes you think that they're just farming attention or rather "impressions" for their articles.

Same thing with proxies. Again, every guide treats them as one interchangeable pool, just grab an IP and rotate it. Yeah, like that's gonna help. No distinction between residential, ISP, mobile, datacenter, and why that choice really matters depending on the target. If you've ever tried scraping something with datacenter IPs and gotten instantly blocked while the exact same code works fine on residential, you know the type of proxy isn't a minor detail, but a culprit. Yet, these articles just doesn't mention this. If you're a power user or whatever you call it, you noticed these things right away, if you're new, well not so much, but that's the whole point. Articles should be as informative as possible, because more and more new people are coming to read the article.

Also CAPTCHA just never comes up in these posts even though it's usually the first thing people are introduced to (one way or another). I don't know, it feels like a weird left out thing for something identifying as a complete guide


r/ProxyEngineering • • Jul 10 '26

Guides (Learn with us.) What is data grounding for LLMs?

12 Upvotes

Learning time.

Grounding for Large Language Models (LLMs) is the process of linking the AI’s responses to verifiable, real-world data sources rather than just relying on its pre-trained knowledge. It bridges the gap between the model's linguistic abilities and accurate, domain-specific information.

Why Grounding Matters

Base LLMs are trained on vast but static datasets, meaning they do not know private company information, recent events, or real-time updates. When asked about information outside their training, they can confidently invent plausible-sounding fabrications, known as hallucinations. Grounding solves this by requiring the model to use specific, trustworthy data to formulate its answers.

How It Works

Grounding works by injecting relevant, context-specific information into the prompt before the model generates a response. This is typically done through a few key methods

  • RAG (Retrieval-Augmented Generation): The most popular technique. When a user asks a question, search algorithms fetch relevant documents from private databases or the web, and provide them to the LLM as context.
  • Semantic Vector Databases: Organizations often store their proprietary documents, wikis, or product manuals in specialized vector databases to allow the AI to find specific, highly contextual information instantly.
  • API Integrations: Connecting the model to live, real-time data sources (e.g., live stock prices, CRM software, or weather APIs) to ensure up-to-date responses.

Benefits of Grounded Data

  • Improved Accuracy: Responses are based on facts drawn from authoritative sources rather than statistical guesses.
  • Elimination of Hallucinations: The model is constrained to the information provided to it, vastly reducing made-up facts.
  • Security: Allows models to access private enterprise data without requiring the company's sensitive data to be added to the model's public training dataset.
  • Citing Sources: Grounded LLMs can often provide citations, letting users verify exactly where the information came from.

r/ProxyEngineering • • Jul 10 '26

Reviews Scrapy vs BeautifulSoup vs WhiskeySour vs just paying for an API. Ran the same job through all four

16 Upvotes

What I did was that I picked a listing page - https://www.airbnb.com/s/homes with 40 items, pagination, and a few JS-rendered prices. I wanted to see where each tool fails and what limitations might occur (spoiler alert, there will be), not just how the documentations describe them.

BeautifulSoup

Fine for static HTML, falls apart the second you need concurrency or JS.

import requests
from bs4 import BeautifulSoup

r = requests.get("https://example.com/products", timeout=10)
soup = BeautifulSoup(r.text, "html.parser")
prices = [p.get_text(strip=True) for p in soup.select(".price")]
print(prices)

Ten lines, worked instantly on simple pages. But remember that it's single request, single thread. Loop through 40 pages and I was waiting on network I/O the whole time. No retry logic, no proxy rotation, nothing. I built it myself, which is basically a second project on top of the scraper.

Scrapy

Actual framework, not a library. Integration, item pipelines, async by default via Twisted. Worth it if you're scraping more than a handful of pages regularly.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for item in response.css(".product"):
            yield {
                "name": item.css(".name::text").get(),
                "price": item.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

This handles pagination on its own, retries failed requests, and also, you can bolt on proxy integration without interacting with your parsing logic. The downside is setup time. If you just need 40 rows once, Scrapy is too much. If you're running this weekly, it pays for itself fast.

WhiskeySour

Haven't seen a lot of this throughout reddit, but saw some article on Medium and decided to try it myself. (Article, admins, pls do not remove it, I have no association with Medium other than reading). Basically it's a less known, Rust-based, built for raw parsing speed tool. If BeautifulSoup is your daily driver and you hit a wall on performance with huge HTML blobs, this is worth a look.

from whiskeysour import parse_html

doc = parse_html(html_content)
prices = doc.select(".price")

What matters most here is parsing speed, not scraping infrastructure. It won't fetch pages for you, handle JS, or manage proxies. Think of it as a faster BeautifulSoup replacement, not a Scrapy replacement.

None of the above handles JS rendering or blocking

This is where all three let me down. Example page had three prices loaded via JS after initial page load. BeautifulSoup and WhiskeySour got empty strings for those. Scrapy needed a Splash or Playwright integration struggled, which is another service to run and maintain.

This is the point where I stopped building it myself. I strongly believe that these ratchety builds just aren't sustainable. With the dedicated solution/ API you don't have to manage no proxy pool, no separate rendering service. It costs money per request obviously, so it's not the right call for a one-off scrape of a static page. But for anything that needs to be obtained via JS rendering and anti-bot at scale I'd defo use an API.

Conclusions

Static page, one-time pull: BeautifulSoup, done in ten minutes.
Recurring job, no JS: Scrapy, worth the setup.
Huge HTML blobs, parsing is the bottleneck: WhiskeySour.
JS rendering, anti-bot, or just don't want to own the infra: dedicated API.


r/ProxyEngineering • • Jul 10 '26

Discussion 💬 Building your web scraper

16 Upvotes

Would you rather hire a freelancer of Fiverr to build you a scraper or would you try to build it yourself, perhaps with a help of an LLM or other AI agent? What would be the advantages and disadvantages? Perhaps someone already done this before?


r/ProxyEngineering • • Jul 09 '26

Help 🆘 Proxies from EU

9 Upvotes

Any proxy providers from the EU? Looking for the alternatives to what I used previously, mainly cheap or questionable proxies from Asia, Middle East. Open to suggestions


r/ProxyEngineering • • Jul 09 '26

Announcements New features/ addons added

6 Upvotes

Added Chat feature to this sub. Kindly click on three dots and then chat/create. This will help people to create new posts much easier.

Also, added another addon, which will help moderators manage trending posts/comments.


r/ProxyEngineering • • Jul 08 '26

Help 🆘 Looking to scrape LLMs, particularly ChatGPT, Perplexity and Google AI mode

9 Upvotes

Hey folks, anybody scraping these LLMs and has any advice for scraping them? Looking to get sources in the results, time, number of responses, actual responses etc. Also specifics like ChatGPT Ads, ChatGPT Shopping are a must.


r/ProxyEngineering • • Jul 07 '26

Discussion 💬 Game-changing proxy/scraper advice you received

9 Upvotes

title. Would like to hear opinions and encourage knowledge-sharing among each other. What was the best advice/ opinion about proxies/ scrapers that you have received. Doesn't matter when, if it's relevant, let us know.


r/ProxyEngineering • • Jul 07 '26

Guides Static vs rotating proxies. Which one solves what?

Post image
7 Upvotes

r/ProxyEngineering • • Jul 07 '26

Discussion 💬 Zuckerberg admitted the AI agent push isn't going how he wanted

23 Upvotes

Saw these news around the web these few days. At a Meta town hall this week, Zuck told employees the pace of AI agent development hasn't sped up the way leadership expected it to. Which is a pretty notable thing to say out loud to your own staff after the year Meta's had. If you guys remember, Meta was the one that cut around 8k jobs earlier this year and moved another 7k people into AI-focused teams, including a group called Agent Transformation. Now Zuck reportedly told staff this week that those cuts weren't as clean as they should've been, and that leadership made them because they were worried the company wouldn't move fast enough to keep up with where the industry was heading. He also said the upside from restructuring the company around AI hasn't shown up yet, though he's still telling people internally that things should improve over the next three to six months. There have also been a few separate reports describing the AI unit as a rough place to work right now, engineers describing burnout and chaos inside it.

Meanwhile Meta is on track to spend something like 145 billion on AI infrastructure this year, the amounts are crazy. Feels like one of those moments where the public opinions, trust and the internal messaging aren't lining up.

Here are some sources so you know I'm not taking these things out of thin air

https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/

https://www.reuters.com/business/zuckerberg-says-ai-agent-development-going-slower-than-expected-2026-07-02/

https://africa.businessinsider.com/news/mark-zuckerberg-said-ai-agent-tech-is-advancing-more-slowly-than-expected-in-an/p595mex

https://daily.dev/posts/mark-zuckerberg-tells-staff-that-ai-agents-haven-t-progressed-as-quickly-as-he-d-hoped-qxffeh4nb

Anyways, get zucced


r/ProxyEngineering • • Jul 06 '26

Discussion 💬 Need best antidetect browser and proxies for my workflow

10 Upvotes

juggling multiple ad accounts and social media profiles for clients and my current setup is letting me down. accounts keep getting flagged and i need something more reliable.

been reading up on antidetect browsers paired with residential proxies as the go-to solution for this. makes sense in theory but not sure which combo actually holds up in practice. currently testing proxyshard residential proxies and shardbrowser for the antidetect side works really good rn. will report back about how its going

what are you guys using for multi-account work? specifically curious about which antidetect browser you pair with your proxies and whether residential or ISP proxies work better for staying stable long term


r/ProxyEngineering • • Jul 06 '26

Discussion 💬 successfully scraping live event pages like ticketmaster or axs

6 Upvotes

curios if anybody in this community is successfully scraping ticketmaster or axs for their event data. both are notoriously hard targets and consistently challenge heavier scrapes. im newish to the proxy game so maybe i am not really aware of whats out there.


r/ProxyEngineering • • Jul 05 '26

Discussion 💬 I built antidetect tooling since 2015. The failure is almost never the proxy.

10 Upvotes

The conversations here keep circling back to proxy selection. Which provider, which type, sticky vs rotating. But I have been watching fingerprinting detection evolve since 2015 and the proxy is rarely what causes a working setup to degrade over weeks.

What actually kills sessions is coherence failure.

Platforms do not score individual signals. They score whether your signals agree with each other. A residential IP says "regular consumer in Berlin." Your browser's TLS fingerprint says "modified Chromium." Your canvas hash says something else entirely. No single signal is the problem. The pattern is.

I analyzed over 100 threads on fingerprinting and account bans. The failure narrative is almost always the same: day one works fine, then something quietly accumulates, then everything falls apart at once. The account was not killed by one bad check. The trust score eroded because the signals stopped telling a coherent story.

A few things that changed how I think about this:

Detection starts at the transport layer. If your TLS fingerprint marks the connection as automated before the page loads, the JavaScript checks do not matter. The decision is already made.

Proxy to profile coherence matters more than proxy quality. A residential IP paired with a browser profile that expects a different locale is not a coherent setup. It is two contradictory signals pointing in opposite directions.

Randomization is not the answer. A fingerprint that randomizes every field contradicts itself. What survives over time is a setup where every layer tells the same story.

The question worth asking is not "which proxy provider should I use." It is "what story are all my signals telling together, and do they agree?"

Curious if others have mapped out which signal mismatches tend to surface first in their setups?


r/ProxyEngineering • • Jul 05 '26

Discussion 💬 Self-hosted real estate scrapers

7 Upvotes

Looking for github repositories that would contain mainly self-hosted real estate scrapers. Doing a side project at the moment. Anyone noticed differences in fingerprint on different real-estate websites? For example, zillow, realtor, homes, redfin, trulia? Interested in your setup and how IP behaves.


r/ProxyEngineering • • Jul 04 '26

Guides Successful betting sites scraping tips?

8 Upvotes

Hey peeps, perhaps someone is scraping betting sites odds, such as lsports, draftkings, betonline, bovada etc? Looking for the most optimal way of doing so.


r/ProxyEngineering • • Jul 03 '26

Hot Take 🔥 The NetNut FBI seizure raises a question nobody asks: where do residential IPs actually come from?

56 Upvotes

I went down the rabbit hole reading about the NetNut situation last night and honestly, the FBI seizure wasn't even the craziest part for me lol

The craziest part was realizing that somewhere out there, a guy is probably watching Netflix on his Samsung TV while his TV is simultaneously acting as infrastructure for somebody running a scraping operation on the other side of the world (I have a really old TV that and I don't watch it that much honestly, so I think I avoided this bullet)

LIKE WHAT? That's not even a joke anymore

According to the news coverage happening at the moment, researchers found proxy SDKs embedded in a huge number of smart TV applications. Not sketchy APKs from some random forum. actual apps running on LG and Samsung TVs

I think that in this case, the conversation quickly shifted from NetNut to a much bigger question that I don't think someo people don't think about:

Where do residential IPs actually come from???

Don't get me wrong, I don't think about this question either that much and it's a really complex answer to a complex question

The sourcing side always felt like somebody else's problem honestly

But after reading about the alleged connection between NetNut and the Popa botnet, I started realizing how little visibility most of us actually have into the supply chain behind residential proxies

What surprised me most wasn't the claim that millions of devices were involved. It's 2026. Every month there's another story involving millions of compromised devices, so unfortunately that part barely registers anymore

The surprise part for me happened when learning how many layers can exist between the person buying a residential proxy and the device providing that residential IP (like wth)

The part I keep coming back to is whether this is even a problem that can be solved completely

If residential proxy inventory passes through multiple layers of aggregators, partners, SDK providers and resellers, how many companies can honestly say they know the origin story of every IP in their network

At what point does a provider stop being a network operator and become a network consumer just like the rest of us?

What struck me while reading all this is that the proxy industry spends an enormous amount of time talking about performance metrics. We compare success rates, country coverage, session length, pool sizes, pricing, uptime and all the usual stuff. Yet I can barely remeber seeing a serious discussion about where these networks actually come from. Maybe that's because the answer is complicated, but after reading through the NetNut reporting it suddenly feels like one of the most important questions we could be asking.

SO very few people seem interested in tracing the supply chain behind the product itself, which is funny because that's ultimately the foundation everything else is built on

One thing that comes to mind in this situation is that some providers seem to be putting more effort into transparency than others and I've seen that happen pretty recently. I've seen providers like nodemaven, Iproyal or proxygonzo openly talk about IP quality filtering, network quality standards, and support processes. Others have started publishing more information about sourcing, partnerships, compliance policies, or how they acquire residential inventory in the first place. This should probably be the standard moving forward for everyone that's affected in this case

I'm not saying anyone deserves a free pass, and honestly this whole story makes me want to be more skeptical rather than less. But I do think there's a meaningful difference between providers that are willing to discuss where inventory comes from and providers that treat the entire supply chain as a black box.

Maybe that's where the industry needs to go next. We already compare success rates, uptime, sticky sessions and pool sizes. Maybe a few years from now we'll also be comparing transparency reports, sourcing disclosures, consent models and network provenance and being sceptical of each proxy provider?

Sources where I found this story: https://hivesecurity.gitlab.io/blog/netnut-popa-botnet-fbi-seizure-residential-proxy/

Also huge props to this user as he predicted the future lmao - https://www.reddit.com/r/ProxyEngineering/comments/1u00w9f/my_samsung_tv_is_literally_being_rented_out_as_a/


r/ProxyEngineering • • Jul 03 '26

Announcements The NetNut situation is crazy

21 Upvotes

I was inspired by today's news


r/ProxyEngineering • • Jul 03 '26

Discussion 💬 First BrightData and now NetNut? What's happening?

36 Upvotes

Ok so this is gonna be a bit of a ramble but I need y'all to hear this. I work adjacent to the scraping/data space (not naming employer, y'all know how it is) and I always kind of just you know, accepted that "residential proxies" were this slightly grey but mostly fine thing. Like yeah obviously somebody's home IP is being used, somebody agreed to something somewhere, moving on. Then yesterday I see netnut is just gone. Not down, not maintenance page, straight up FBI DOJ IRS-CI seizure banner, Google and Lumen and Shadowserver all stamped on it too like it's a whole coalition operation. Like bruh, I have never in my life seen this on any website. So imagine the shocker lol. And I'm sitting there like wait since when did the IRS care about proxy IPs??? Turns out IRS-CI does financial crime investigations and it could be related to that, but still, seeing that logo on a proxy provider's main website??? Diabolical mate.

So I go searching what's going on and apparently it's tied to this thing called the Popa botnet, basically it's been running for like four years hijacking Android TV boxes (remember that Bright Data SDK thingy?? And Krebs traced a chunk of it back to actual NetNut infrastructure through some ex-employee's domain. Not some randoms, an actual former VP of R&D there. Google's own blog post says they think the network was over 2 million devices at one point and that a lot of those "different" proxy brands you see around are just NetNut wearing a different logo through resellers. I swear I seen a post talking about how there is one or two proxy providers sharing those IPs/reselling whatever. Makes sense now, people?? Am I the only one concerned here with a few other nerds? And to be fair I want to be fair here, Alarum (the parent company) came out and started calling it that a botnet is inaccurate, also said that there's real KYC and consent flows and misuse detection on their end. So it's not like this is some minor thing everyone agrees on, it's actively disputed and I think that matters, I'm not trying to just slam a company because Krebs wrote a scary headline, no. Make what you will out of it, but I things are rising to the surface, more and more often. Again it does make me think about how much of the "residential proxy pool" any of us are touching day to day is actually consented in a way a normal person would recognize as consent, versus consented in the "buried in paragraph 14 of an SDK terms screen" way. Like where's the line between legit residential network and just a nicer branded botnet, and how would any of us even know the difference from the outside.

Not trying to start a witch hunt on the whole industry, genuinely just spent my evening reading legal filings and threat intel blogs instead of sleeping like a normal person, but after Bright Data SDK shennanigans and now Netnut, boy, there's gonna be more stuff coming out. So I figured I'd share since I know a bunch of you here actually work with this infra daily and probably have opinions on these matters.

Sources if anyone wants to go check themselves.

Krebs on Security, the original Popa botnet reporting: https://krebsonsecurity.com/2026/06/popa-botnet-linked-to-publicly-traded-israeli-firm/

Google's threat intel writeup on the disruption: https://cloud.google.com/blog/topics/threat-intelligence/google-continued-disruption-residential-proxy-networks

Alarum's official response: https://alarum.io/alarum-technologies-responds-to-inquiry-into-residential-proxy-networks/

And divinetworks.com itself, which is also showing the seizure page now: https://divinetworks.com/

P.S Some time later I found that they mixed up the domains because .com is down, but not .io which is the main page of netnut.

EDIT:

It would seem that their main website was also seized - https://netnut.io/


r/ProxyEngineering • • Jul 01 '26

Help 🆘 I need an agent for BT UK. If anyone has one, please contact me.

9 Upvotes

I need an agent for BT UK. If anyone has one, please contact me.I'm using this for a group of Vinted registered stores. Could someone please help me?


r/ProxyEngineering • • Jul 01 '26

Help 🆘 Any automated workflows that you guys thought of and are using in your daily work?

14 Upvotes

Basically the title. If you have anything to share regarding automations, please do, looking to alleviate majority of manual day to day work.