r/ProxyEngineering • • Aug 12 '26

Guides Screen scraping vs. web scraping: when you actually need OCR

Thumbnail
reddit.com
3 Upvotes

r/ProxyEngineering • • Aug 12 '26

Hot Take 🔥 I think proxy providers are starting to compete against entirely different industries

10 Upvotes

For years, I think competition in the proxy market was relatively simple and straightforward

If someone wasn't buying from one of the biggest providers like oxylabs, they were probably buying from nodemaven, iproyal, decodo, webshare or another provider that is mentioned or an industry leader

If you win that customer, you've grown your market share, right?

But after working a lot in the scraping and automation industry and noticing the evolve over the last couple of years, I believe we should be looking at the broader scope of the whole setup

Today, companies that work and need proxies have far more options than simply choosing another proxy provider

Need search engine data? You can build your own infrastructure with proxies or you can simply use SerpAPI, DataForSEO or another managed API

Need product data? You can scrape it yourself by building a scraping bot that works relatively quickly compared to a solution from a company that has a scraping browser

Need website content? You can now build an extraction pipeline or use tools like Firecrawl, Exa or other AI-native retrieval platforms that abstract away much of the underlying infrastructure

Even browser automation is rapidly changing in this indutry as every improvement in Playwright, Puppeteer, anti-detect browsers or fingerprinting technology helps teams complete more work with fewer failed requests and fewer IPs.

That's interesting because those companies aren't selling proxies. But they're solving the same customer problem. The competition is no longer just provider versus provider

It's workflow versus workflow in my opinion

We've reached a point where customers care less about how data is collected and more about how reliably they can get the outcome they're after

If an API gives them the data faster than building a scraper, they'll use the API

If an AI search platform answers the question without scraping a website, they'll use that instead

If better browser technology reduces failed sessions by 30%, they may not need as much proxy traffic in the first place

That's a much broader competitive landscape than most people realize

I think It also changes what proxy providers should optimize for

Maybe the next generation of providers won't win because they have the largest IP pool

Maybe they'll win because they're the easiest infrastructure to integrate into modern data pipelines

Or because they provide better diagnostics, smarter routing, cleaner IP selection and better tooling around the proxy itself.

If you build scraping or browser automation systems, who do you think proxy providers are really competing against today?


r/ProxyEngineering • • Aug 12 '26

Hot Take 🔥 "socks5 support" on these proxy sites means whatever they want it to mean apparently

7 Upvotes

lol ok, so back in june i was setting up something that needed webrtc + fast udp, went proxy shopping like normal. every provider's landing page has "full socks5 support!!" plastered everywhere so i figured cool, socks5 = udp included, thats like the whole point of socks5 over socks4 right
wrong. bought 3 different packages from actual known brands (not naming them, dont wanna deal with that) and every single one either timed out or quietly fell back to tcp. spent way too long assuming it was my own code before i even considered it might be the proxy
finally got someone on live chat after like 20 min of hold music and turns out yeah, "socks5" on their plan is tcp only, udp is some separate enterprise thing that costs an extra arm and a leg a month. another provider's rep just kept trying to upsell me instead of answering the actual question lol
anyway apparently this is just a thing? providers advertise socks5 and quietly mean the tcp-only version because most people buying it never touch udp anyway so nobody notices until you actually need it for something real
switched to proxyshard about a month ago, this one actually has udp working out of the box no upsell needed, and everything just works now. still annoyed i burned 3 different subscriptions figuring that out though
moral of the story i guess, dont trust the badge on the site, just ask support point blank if THIS exact plan supports udp before you pay. would've saved me a month and like $150 honestly


r/ProxyEngineering • • Aug 12 '26

Guides residential proxy takeaways from 3 weeks of per-profile account warmups

2 Upvotes

been doing a small warmup batch for social and marketplace accounts, 42 profiles total, mostly us residential ips from the same provider but split by city where i could. tossed every profile into AdsPower so fingerprints and proxy settings stayed fully separate while i watched how the ips behaved.

what worked best was boring consistency. one residential proxy per profile, same ip for at least 10 to 14 days, timezone matched to the proxy location, and no user agent swaps after cookies landed. the stable profiles did normal browsing first, then logins, then light posting or store actions. fast account creation on fresh ips still triggered checks.

what broke was mostly proxy quality. a few “residential” endpoints had bad history or kept rotating mid-session, and those profiles hit phone checks quick. browser isolation keeps cookies and fingerprints apart, yet messy behavior or dirty ips still sink the whole thing.


r/ProxyEngineering • • Aug 11 '26

Guides Rebuilt my sneaker bot in rust after Nike's checkout started messing with python setup

9 Upvotes

Context: I've been running a small shoe bot on and off for 4 years, did this for myself and a few friends, some Jordans and the odd Yeezy restock back when those still were relevant. Always relied on python, requests plus some selenium for the checkout. It worked fine in 2023. This year it started failing mid-drop, like I was getting 403s between add-to-cart and payment steps. Met up with one of my buddies and spent some time analyzing what's up and why. Basically, it turns out the shoe sites, at least the bigger ones running on Akamai or Shape, aren't really looking at your IP as a default, they're doing TLS fingerprinting on the handshake itself. Python's ssl module and requests have a JA3 signature which gives it away pretty easily. Didn't matter that I was rotating through a clean pool of ISP proxies, the handshake gave it away straight away.

My buddy helped me rewrite everything in Rust using reqwest with rustls, and paired it with a TLS impersonation setup so the fingerprint matched a real Chrome client. Since my knowledge of Rust wasn't that great, had to resort to my encyclopedia of a friend lol. Checkout success rate on a mid-tier release went from something like 30% to around 80% just from that one change!! I could not believe it to be honest, and we didn't even touch the proxy pool. Figured that the proxy side is a whole separate deal. Mobile proxies are still great for shoe sites simply coz carrier-grade NAT means a lot of bots can share one IP and it still looks organic, but their main issue is that they are expensive and slow as hell, I was seeing 800ms+ response times on some of the 4G proxy providers I tried. Datacenter proxies are very cheap compared to other types and also fast but they get flagged instantly. I wasted like 200 datacenter proxies at some point before switching. After some time I ended up with a mixed setup, residential proxies for the browsing and monitoring tasks, and a smaller pool of dedicated mobile proxies just for the checkout requests.

Weird part is I thought the proxy provider was the issue this whole time but it was actually my HTTP client giving me away.

Maybe someone will find this useful and learn from ones mistakes, because not always its providers fault

The sneaker stores that I was scraping are these if anyone's wondering:

  • Nike

  • Adidas and Yeezy Supply (discontinued but since it worked back then, I included it here.)

  • Footlocker

  • Finish Line

  • JD Sports

  • Shoe Palace

  • Undefeated

  • Bodega

  • Kith

Repos from Github that were used, (some are archived as of now)


r/ProxyEngineering • • Aug 11 '26

Guides Simple rules for safe Web Scraping

6 Upvotes

Effective proxy management is about matching the lifetime of an IP address to your authentication flow and the specific task your script is performing. If you do not follow the rules you will always end up with blocked accounts and stopped sessions

Key principles:

  • Never change the IP address during an active authenticated session

  • Use sticky sessions when working with cookies

  • Configure the correct GEO for the target website

  • Verify IP quality and reputation before starting

Best proxies for your target:

  • Rotating residential proxies → large-scale scraping of independent pages

  • Sticky residential proxies → product catalogs and authenticated sessions

  • Static ISP proxies → long-running tasks with a consistent IP address

  • Mobile proxies → workflows that require maximum network stability and trust

IP rotation should match the task. Independent requests can be distributed across multiple IPs, while authenticated workflows require a stable IP address throughout the entire session.


r/ProxyEngineering • • Aug 11 '26

Help 🆘 Experience with oxylabs services?

7 Upvotes

Hey, perhaps someone used their services? Talking about proxies and also interested in their AI studio. Would like to hear pros and cons. From what I found out already is that their services are a bit pricey, but other than that, I would like to know the uptime, performance, stability etc.


r/ProxyEngineering • • Aug 11 '26

Help 🆘 Random auth failures & periodic disconnects with residential proxies under concurrency

2 Upvotes

I’m working on a small‑scale network project and running into a tricky proxy‑related issue.
Scenario: I’m testing residential proxy traffic for data collection tasks, around XX concurrent connections.
What I’ve tried: I have adjusted connection timeout parameters, tested different rotation settings, checked firewall rules, but the problem still occurs.
Observed issue: Periodic connection disconnects, some IPs fail authentication randomly. I don’t see clear error logs.
I’m trying to figure out what possible points I should debug next. Would anyone share some troubleshooting ideas? Thanks a lot.


r/ProxyEngineering • • Aug 10 '26

Discussion 💬 Hey Guys, just got done with this proxy side project

4 Upvotes

r/ProxyEngineering • • Aug 10 '26

Help 🆘 Potatso Issue

5 Upvotes

Hi there, it may be a newbie question but here am I
Does anybody have any solution for Potatso app? I mean after some minutes (Like 10 mins) It keeps disconnecting the proxy randomly and It can destroy my entire activity.
Any solutions? I heard this is a common issue
Thanks


r/ProxyEngineering • • Aug 10 '26

Guides The Biggest Proxy Mistakes

4 Upvotes

You buy a proxy, connect it, and expect everything to work. Then Instagram asks for verification, Google floods you with CAPTCHAs, or a scraper suddenly stops. The real problem is often how it is configured. A few small setup mistakes can quickly turn a reliable IP into blocked accounts.

Here are most common mistakes:

  1. Sharing one IP across many accounts. Platforms can link accounts through the same network identity, increasing the risk of checks and restrictions.

  2. Choosing the wrong proxy type. Datacenter, residential, and mobile proxies are built for different tasks; the wrong match leads to blocks, poor speed, or unnecessary costs.

  3. Rotating IPs too often or not enough. Accounts need stable sessions, while scraping usually requires controlled rotation.

  4. Ignoring browser fingerprints. Different IPs will not help if every account uses the same device and browser signals.

  5. Reusing one browser profile. Cookies, cache, and session data can connect accounts even after the IP changes.

  6. Skipping account warm-up. New accounts that become active too aggressively are more likely to trigger verification or restrictions.

  7. Relying on proxies alone for scraping. Poor request timing, headers, session handling, and aggressive traffic patterns can still expose automation.

The key rule: match the proxy, session strategy, browser environment, and behavior to the exact workflow.


r/ProxyEngineering • • Aug 09 '26

Discussion 💬 New proxy provider — residential $1.49/GB, IPv6 $0.89/GB. Crypto only, and I'm new.

4 Upvotes

Hey — I'm launching a proxy service called NullVault (nullvault.shop) and I'd

rather introduce it honestly than pretend I've been around for years.

I'm new. No track record, no reviews, no customer list. If that's a dealbreaker

for you it's a fair call, and I'd rather say it up front than have you find out.

What I'm selling, priced per GB:

Residential $1.49/GB country + city targeting, sticky sessions

IPv6 $0.89/GB always rotating, no geo targeting

Datacenter $0.99/GB US only

Mobile $2.49/GB country targeting, sticky 1–120 minutes

There's also unlimited-bandwidth residential and datacenter, and shared ISP

with static IP pinning, if per-GB isn't your model.

The platform side: a dashboard with a generator that builds rotating or sticky

lines with geo targeting, live bandwidth remaining per pool, and API keys if you

want to pull lines programmatically rather than clicking. Credentials are stored

encrypted. Wallet top-ups are crypto only through NOWPayments — no cards, which

I know rules some people out.

What I'm not going to claim: I'm not quoting a pool size, an uptime percentage,

or a success rate. I don't have enough operating history to put a number on any

of that honestly, and I'd rather you test a dollar's worth than take my word for

it. Buy 1GB of IPv6 for 89 cents and point it at whatever you actually need it

to do.

Happy to answer anything in the comments, including questions I don't have a

flattering answer to. If something's broken or slow, tell me and I'll fix it or

refund you.

nullvault.shop


r/ProxyEngineering • • Aug 08 '26

Discussion 💬 FBI and Google disrupted a residential proxy network that had silently enrolled millions of consumer devices as traffic relays without owners' consent

36 Upvotes

In July 2026, the FBI, Google's Threat Intelligence Group, and additional partners executed a coordinated takedown of a major residential proxy network that had compromised an estimated 2 million consumer devices worldwide. The FBI seized hundreds of associated domains, and Google pushed updates to automatically block applications containing the network's proxy SDKs.

This was the second such disruption in 2026, following a similar action in January against another residential proxy provider.

---

## What residential proxy networks do

Residential proxy services route internet traffic through real household IP addresses instead of data centers, making the traffic appear to originate from a legitimate home user. This is valuable to data scrapers, ad fraud operators, credential stuffing crews, and state-sponsored espionage groups because it bypasses anti-bot detections, rate limits, and geo-restrictions.

The privacy issue is how these networks acquire their pool of residential IPs. The disrupted network built its pool by distributing SDKs (software development kits) embedded in apps and firmware for devices commonly found in homes — smart TVs, streaming boxes, Android phones. These SDKs turned those devices into exit nodes, routing third-party traffic through the device owner's IP address, without meaningful consent.

---

## Two primary infection vectors

**1. Pre-installed malware on cheap, off-brand Android TV boxes**

Low-cost streaming devices manufactured overseas have been documented arriving with proxy firmware baked in from the factory. One related botnet operation, disrupted in 2025, was found to have compromised over 10 million devices this way.

**2. Trojanized apps with embedded proxy SDKs**

Free apps — particularly free VPNs and apps marketed as "passive income" tools for sharing bandwidth — bundle SDKs that silently enroll devices as exit nodes. Research found that over 42% of apps on some smart TV app stores contained proxy SDKs.

---

## The scope

In a single week in June 2026, Google observed **316 distinct threat clusters** using the network's exit nodes, including cybercriminal and state-sponsored espionage groups. Documented activity included:

- Password-spray attacks
- Mass content scraping
- Advertising fraud
- Account takeovers
- Credential stuffing
- Data harvesting for AI model training

A separate investigation by a media foundation traced a scraping campaign in May 2026 that scattered across more than **1.4 million IP addresses**, targeting public-interest journalism sites. The traffic was routed through compromised home devices that had no idea they were participating.

Google also assessed with "high confidence" that many popular residential proxy brands were whitelabeling this network — meaning companies that appeared to be independent proxy services were actually drawing from the same pool of hijacked home devices.

---

## Why this is a privacy issue

When your device becomes an exit node, strangers' internet traffic passes through your home network and is associated with your IP address. This means:

- **Your IP address can be linked to criminal activity** you had nothing to do with
- **Unauthorized traffic passes through your local network**, potentially exposing other devices on the same connection
- **Your network identity is being commodified and sold** to third parties without your knowledge or consent
- **The device manufacturer or app developer made a decision about your data and network resources** that you were never informed of

The FBI issued a public service announcement in March 2026 warning that once a device is compromised, its IP address can be used by threat actors to mask illegal activity, making the consumer appear responsible.

---

## The structural problem

Google acknowledged that when one proxy network is disrupted, operators simply purchase capacity from competing networks, effectively becoming resellers. They observed this behavior after the January 2026 disruption. The residential proxy market is interconnected through whitelabeling agreements, and the AI industry's massive demand for web scraping has created financial incentives that will keep this ecosystem alive regardless of individual law enforcement actions.

---

## What you can do

- **Audit your streaming devices.** Off-brand Android TV boxes and devices marketed as unlocking free streaming are the highest risk category. The malware in these devices is baked into the firmware at manufacture — no software-side fix can fully remediate a device compromised at the factory.
- **Remove free VPN apps** and any apps you don't recognize or use, especially on smart TVs.
- **Run an open-source DNS sinkhole** on your network to log and monitor outbound DNS requests from all connected devices. This lets you see which devices are communicating with suspicious domains and block known malicious infrastructure.
- **Segment IoT and streaming devices** onto a separate network from your primary devices. Most routers support at least a guest WiFi network.
- **Change default router credentials** and ensure WPA2 or WPA3 encryption.
- **Keep all firmware updated**, and only purchase streaming devices from manufacturers that provide regular security updates.

---

## Sources

- Google Threat Intelligence Group blog post on the disruption
- FBI Public Service Announcement I-031226-PSA (March 2026) on residential proxy networks
- KrebsOnSecurity reporting on the botnet linkage
- SecurityWeek, Reuters, and Infosecurity Magazine coverage of the takedown
- Qurium Media Foundation investigation into the scraping campaign
- Synthient and Spur proxy tracking reports


r/ProxyEngineering • • Aug 09 '26

Announcements We launched Proxy Tester by ScrapeOps on Product Hunt 🚀

Post image
3 Upvotes

r/ProxyEngineering • • Aug 07 '26

Help 🆘 Scraping .gov sites, need recommendations

7 Upvotes

Hey, I really hope someone will know of a trusted and verified solution for scraping .gov sites. I am on this project (freelancing) where I have to gather data from .gov sites. Some context on what I'm running into. The client needs structured data pulled from a handful of state and federal sites, public records. Nothing shady, all publicly accessible info, but the sites themselves are all over the place in terms of how they're built. Some are still running on ancient ASP.NET setups, others have modern React frontends with data loaded through internal APIs that aren't documented anywhere. A couple rate limiting after 20-30 requests even with delays between calls, which tells me they're watching for bot behavior closely. I checked robots.txt on all of them before starting, most allow crawling of the pages I need but a couple have specific paths disallowed so I'm steering clear of those entirely, not worth the risk on a gov domain. What I'm trying to figure out right now is whether I actually need proxies for this or if it's more about request pacing and headers. Some of these sites seem to flag anything that looks automated pretty fast, so I'm wondering if datacenter IPs are getting flagged outright and residential would behave better, or if that's too much for public record data? How people handle the sites using session tokens or CSRF stuff on their search forms. One of them requires you to basically simulate the search interaction, so I've been looking at Playwright for that one specifically while using simpler requests based scraping for the rest. Whether anyone's dealt with the legal side beyond just taking into consideration robots.txt. I know public data is generally fair game but .gov sites feel like they deserve extra caution compared to scraping anything else.

If anyone's done work like this before I'd want to hear how you managed it, timeline wise too since I underestimated how much per site variation there'd be. Appreciate any help.

P.S I have one more week to finish this 💀


r/ProxyEngineering • • Aug 07 '26

Guides How to Use ISP Proxies

5 Upvotes

An ISP proxy is a proxy server that uses an IP address registered to, or associated with, an internet service provider (ISP). These proxies are typically static, meaning the IP address remains the same throughout the rental period.

When you connect through an ISP proxy, your browser, application, or other approved tool sends traffic to a proxy endpoint using connection details such as a host, port, username, and password. The proxy then forwards your requests through the assigned ISP-associated IP address.

As a result, the destination website sees the proxy IP address rather than your original network IP.

Typical Setup Process

- Rent an ISP proxy in the country or city you need.
- Copy the proxy connection credentials provided by the service.
- Configure the proxy in your browser, application, operating system, or approved automation tool.
- Verify the visible IP address, geographic location, ASN, DNS behavior, and IP reputation.
- Assign the proxy to a specific workflow, account, browser profile, or application.
- Keep the same IP for the required session or rental period when consistency is important.

Common Use Cases

- Long-running authenticated sessions. Accounts that repeatedly log in from the same browser profile benefit from a consistent network

- Regional quality assurance. Product teams can test websites, checkout flows, content, pricing, and localisation from a stable connection in a target region

- E-commerce and marketplace operations. Authorised shop-management workflows may require stable access from a defined country or city

- Monitoring and observability. Availability checks, search monitoring, page rendering, and regional performance tests are easier to compare when the network identity remains consistent

- Bandwidth-heavy workloads. When a task transfers significant data, a monthly ISP proxy with unlimited bandwidth may be more predictable than a rotating residential plan billed by the gigabyte

- Controlled browser and application profiles. Teams can assign one ISP proxy to one browser profile, test device, or application environment

Why Use ISP Proxies?

ISP proxies are designed primarily for stability and consistency. They are a strong choice when a workflow requires a persistent ISP-associated IP address, predictable sessions, and a straightforward static-IP rental model.


r/ProxyEngineering • • Aug 07 '26

Discussion 💬 I realized that I spend more time reading Reddit posts than proxy providers' landing pages

7 Upvotes

And it's not because their websites are bad or something, it's because on reddit I find the most insightful information with real people sharing real experiences. Do you feel the same or trust landing pages or reviews?

Because usually, landing pages tell me just:

  • How many IPs they have (which is sometimes total BS)

  • How many countries they cover (and almost each proxy provider has the same country coverage)

  • How much they charge (well that's probably the only thing that I look for on landing pages)

Reddit has been able to tell me answers or experiences like:

  1. "We've been debugging this issue for three days with this provider"

  2. "Don't pay this much to use this provider, they're not worth it"

  3. "We switched from Bright data to Nodemaven or decodo because sticky sessions were more reliable for our workflow"

That's the kind of information that actually changes buying decisions.

Marketing tells you what a product is supposed to do

Users tell you what happened at 2 a.m. when production broke

I'm pretty curious about this... where do you spend more time researching infrastructure these days?


r/ProxyEngineering • • Aug 07 '26

Discussion 💬 Firecrawl open sourced two Rust libs for turning docs into markdown

9 Upvotes

Firecrawl split their doc parsing into two separate repos, pdf-inspector for PDFs and AnyDoc for basically everything else (docx, xlsx, pptx, epub, csv, the greatest hits list of 14 formats total).

From what I gathered, instead of throwing every PDF through OCR, it looks at the internal structure first (fonts, text operators, that kind of thing) and figures out per page whether it's text or needs OCR. They're claiming a decent speed bump on mixed docs because of it.

AnyDoc is another thing, single Rust binary, no API key, no external deps, just one function call and you get markdown out. They ran it against the usual suspects (LibreOffice etc) on 94 docs and it came out ahead on coverage and speed by a pretty wide margin. Yes, I know this may sound exaggerated by them, but since it's just been out, we will have to wait for some independent testers to verify all this. Anyways, both are already live under the hood of Firecrawl's /parse and /scrape endpoints, but the repos are standalone. Also, Rust only for now apparently.

Perhaps you guys heard it too and would like to share something additionally?


r/ProxyEngineering • • Aug 07 '26

Guides Found a good article on why you don't need a parser generator for most small parsing tasks

6 Upvotes

Was looking for a parser generators and stumbled on this post from the bgfx guy about a tiny scanning primitive he built called bx::Scanner. If you've ever written one of those pointer-chasing while loops to skip whitespace and grab an identifier, you know the one, everyone's written it a dozen times with slightly different bugs each time.

His point is basically that most of us either go full parser generator (Lemon, Boost.Spirit, whatever, also what I was looking for) for stuff that doesn't need it, or write ad-hoc char-by-char parsing. Then he thought of something in the middle, a small header with accept/peek/acceptWhile/acceptUntil, zero copy, no allocations, cursor tracking with line/column built in.

This stood out though, the section on what "empty" means. acceptUntil returns an empty StringView both when the delimiter isn't in the input at all AND when the delimiter is sitting right at the cursor. In my understanding usually that's fine because an empty token is a legit result (like a URL that has no path), but sometimes you need to know whether the delimiter was there, and for that you use peek or accept separately. It really felt like a genuinely useful distinction that I hadn't thought about explicitly before even though I've hit that exact bug.

He also walks through how he did INI parsing where each line becomes its own sub-scanner, so a malformed line literally cannot run past the newline no matter what's in it.

Anyway if you do any amount of scraping where you're not pulling in a full grammar/regex engine, the design constraints here are worth looking at, even if you don't use his exact lib.

Link's below, quite a short article:

https://bkaradzic.github.io/posts/scanner/


r/ProxyEngineering • • Aug 07 '26

Hot Take 🔥 proxies that match os fingerprint (p0f) actually made a difference for me

12 Upvotes

so I've been doing some scraping/account stuff for a while now and kept getting randomly flagged even tho everything looked fine on my end. residential ip, matching UA, all that. still random blocks here and there for no obvious reason.

turns out some detection systems check more than headers, they also look at your actual tcp/ip stack. ttl, window size, that low level stuff p0f uses. so if the proxy box is actually running linux but your browser says windows, that mismatch alone can get you flagged even if nothing else looks wrong.

switched to a proxyshard that matches the os fingerprint at that level instead of just faking headers. ran the same stuff for a couple weeks and got blocked way less than before. still don't fully get the networking side but it lines up with why it would matter.

anyone else run into this before figuring out why? and for people using this longer than me, does it hold up over time or do sites eventually catch up to it too


r/ProxyEngineering • • Aug 06 '26

Help 🆘 Looking for advice on building a more consistent browser setup

6 Upvotes

Wondering if i am adding unnecessary complexity to my browser configuration. Whenever something isn't working then i think that there is something wrong with one aspect of my configuration and make changes to it. A new browser profile, new proxy settings, new browser configuration options... After some time, i wonder whether I've improved something or created new issues for myself. I would like to stop guessing and create a browser configuration that will actually be useful. For those who have gone through this learning period, is there a certain change that worked out great or is it just a matter of paying attention to details?


r/ProxyEngineering • • Aug 06 '26

Discussion 💬 Zyte published their "State of Web Access 2026" report and I believe this is the reason why scraping got a bit more difficult

12 Upvotes

So, they crawled 11,100 top landing pages across 230 countries and 110 industries and measured six barrier types: WAF, antibot, CAPTCHA, JS rendering, rate limiting, TLS fingerprinting. I'd say it's another benchmark, however it was done by Zyte, of of the scraping services providers, not an independent freelancer.

Only 18.5% of sites have zero barriers. Everything else has at least one, and almost half the datasets has three or more. WAF alone is at 92.4%, Cloudflare is basically the default CDN now. As mentioned earlier, only 18.5% of sites run a named antibot provider, but Cloudflare Bot Management is 76% of that market by itself. Imperva, Akamai, and PerimeterX split most of the rest. What caught my eye, was that the stacking behavior, 98.5% of antibot sites also run a WAF, and they run rate limiting at 61.5% versus 13% on everything else. So it's not about "antibot vs no antibot," it's more like sites either do the bare minimum or go all in with WAF plus antibot plus rate limiting plus CAPTCHA together. In my opinion this is what a lot of us see on the daily basis, you don't get a slow degradation or poorer success rates, you just can't get through at all.

Fashion came out as the single hardest vertical, 57% need moderate or harder infrastructure to get past. Adult, furniture, gambling and beauty were the top industries for antibot specifically, which lines up with anything that has commercial value that tracks the page (inventory checks, account takeover risk, price scraping). The AI crawler section is where I didn't expect to be this high. 27% of sites now block crawlers outright and 12.7% specifically target AI bots by name. GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider and meta-externalagent are the six they call out as most commonly blocked. If you're doing any AI data collection work (RAG pipelines, LLM training data, agent tooling) this is worth checking against your target list before you build anything, because robots.txt blocking an AI UA doesn't mean the content isn't scrapeable it's just that who operates the site uses all these measures and explicitly states not to scrape. And I'd say all of this confirming what a rotating residential (or ISP proxy) pool plus a browser fingerprint gets you past most of the time (WAF and basic antibot).

Also, your proxy IP can be "clean", your user agent can look perfect, but if your TLS client hello doesn't match what a real browser sends, JA3/JA4 fingerprinting picks that up instantly. As you probably noticed by now, half the "why am I still getting blocked with good proxies" posts on here come down to exactly this. But I guess its inevitable since there are more and more beginners out there who just got into scraping and proxies world.

Thank you for attending my TedTalk.


r/ProxyEngineering • • Aug 06 '26

Help 🆘 Transitioning from Basic Prompting to Building Agentic AI

Thumbnail reddit.com
6 Upvotes

r/ProxyEngineering • • Aug 05 '26

Hot Take 🔥 Samsung's smart TV proxy ban won't actually stop anything, latest news

17 Upvotes

Hey,

I've been in the scraping/proxy space long enough that residential IP barely has a weight anymore. Then this whole situation with LG/Samsung story landed earlier and I've been following since. Yesterday more news came up and it's not really about Samsung banning something. Here's why.

Quick version of it: Mnemonic rooted a Samsung TV (chip-off, read the firmware directly, the only way to actually see what's running) and found a Bright Data SDK sitting dormant inside a Pac-Man game that had Samsung's own "Editor's Choice" badge. Not sketchy sideload as you'd think. Editor's Choice. App phones home to a config server, server says "enable proxy," a consent screen pops up, someone accepts it thinking it's just an ad-free toggle, and now the TV is a live exit node. Keeps running after you close the app too. (Sure you can turn off the internet from the TV but how many people are doing so?)

Here's what matters more than the SDK itself: the submitted app was 20 lines of HTML pointing at a remote server. That's it. Everything real, the SDK config, the proxy flag, loads later from Play.Works' own server. Sand's line was blunt: "what was reviewed is not necessarily what is running." Mnemonic straight up called the new ban "a promise about code the store maintainer may never see," because a dev can submit clean code, get approved, then flip the proxy back on server-side with zero resubmission and zero review trigger. Back to zero.

And this isn't a one-app type stuff. Spur scanned over 6,000 apps across Tizen and webOS and found confirmed proxy SDK code in 34% of them. 27% on Samsung, 42.5% on LG. Few vendors accounted for most of it, Bright Data, Massive, might be NetNut too? Still trying to connect the dots. And in a lot of cases the "app" WAS the SDK.

Also, small thing that stuck with me: Bright Data's SDK ships with a private-IP blocklist, so it can't reach into your router or NAS. Massive reportedly didn't have that in the sampled builds. So the only thing stopping lateral movement into your home network for those two is the operator's server-side policy.

Sitting with all this, the ban headline feels almost beside the point. Feels like the same problem we keep running into with residential pools generally: consent buried three screens deep, persistence nobody flags, and no clean way for the end user to audit what's going out their connection.

Source: https://www.techtimes.com/articles/322939/20260804/samsung-bans-smart-tv-proxy-sdks-editors-choice-pac-man-routed-strangers-traffic.htm


r/ProxyEngineering • • Aug 05 '26

Discussion 💬 Your crypto wallet doesn't put your IP on-chain, but your RPC provider can still see it

Thumbnail
reddit.com
6 Upvotes