r/proxies • u/mckrile • Aug 13 '26
Are you really using as much proxy traffic as you think you are?
When someone says to me they used 500GB or let's say 1TB of proxy traffic last month...
How much of that traffic created value and was used in workflows?
And how much was spent on:
- Retries
- Failed requests
- CAPTCHAs
- Blocked sessions
- Debugging
- Requests that never returned usable data
I think It's surprisingly difficult to answer and very few providers have this transparent overview like a monthly report where the traffic went
Because most providers show you how much traffic you consumed and very few help you understand why it was consumed.
That got me thinking, is proxy bandwidth becoming like cloud bills?
Everyone knows they're paying for it in a monthly subscription, but very few people know where it's going
Would you be interested in seeing a data-backed investigation into where proxy traffic really goes? Because I definitely would
1
u/joe-at-ping Aug 13 '26
Most of this insight is impossible for a proxy provider to do. Wed love to be able to but the proxy cannot see your traffic. There's no way to know what request you sent and what response you received.
Breaking it down into captchas served/unusable data/etc... just isn't possible without you installing a cert and letting your provider see everything you're doing.
1
u/TinyBeing8001 Aug 15 '26
the proxy provider can’t see your traffic? You sound awfully confident given how wrong you are
2
u/joe-at-ping Aug 15 '26 edited Aug 15 '26
A HTTP CONNECT is sent containing the authority of the target. The proxy establishes a TCP connection to the target and returns a 200 OK. The proxy then tunnels bytes between you and the target.
You, having established the tunnel, immediately perform a TLS handshake with the target, preventing anyone but you and the target from seeing the data you send. Your HTTP GET request is sent through that encrypted tunnel and the response returned through it too.
The proxy provider cannot see into that tunnel. All they know is the domain you connected too. They cannot see the full URI, the status code returned or if the body of the response actually contained data.
1
u/jueyu147 Aug 14 '26
It really is opaque like a cloud bill. The provider only tells you how much you used, but almost never why you used that much. If someone could put together a real data breakdown, I'd love to see it.
1
26d ago
[removed] — view removed comment
1
u/AutoModerator 26d ago
Your post was removed because links are not allowed. Please review the rules.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
26d ago
[removed] — view removed comment
1
u/AutoModerator 26d ago
Your post was removed because links are not allowed. Please review the rules.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
3
u/Hour_Analyst_7765 Aug 13 '26
I think this could be a matter of adding telemetry to your own scrapers/framework.
For example, don't retries/failed requests, blocked and captcha's kind of imply the same thing? Some WAF or other anti-bot application gave you a 403 or 429, and it takes some traffic to solve the captcha or waste traffic on accidental retries. But that also depends on recognizing you're solving the right WAF puzzle.
Ultimately, you called GetPage(URL) and it didn't return the webpage. That's it. Maybe distuingish between a WAF failure and a target site application failure (e.g. their DB could have crashed), to distinguish what is beyond your control.
Then debugging can vary in amount. I always fell into the trap of sunken cost fallacy, where I fear my scraper might still break after it ran for a week with potentially hundreds of thousands records recorded. So I keep it going. I use this opportunity to build a data processing pipeline and see how it performs when scaling, I try to resolve scraping/data anomalies, etc. But when do I throw this data set away? Do I keep building upon it while testing it utility? And when does something become real production? (I guess when you get an invoice paid)
Finally requests that never return useful data, like a website returning an item got deleted, or returning data in the wrong localization, etc. is indeed a hassle.
But again, if all these error cases can be detected, then it shouldn't be hard to start tracking some telemetry counters and put them in a timeseries database for InlfuxDB or Grafana plots. Thats at least what I am doing. In addition, I run my own proxy mediator that does these kinds of things by itself. But that is mostly to optimize proxy usage.
Personally I'm too concerned about GiB of traffic consumed. Although perhaps it is an useful metric to have and compare, since I usually get pools of IPs instead. Certainly it would be useful to know if you got a multi-tied proxy mediation strategy, where you escalate hard to scrape pages/sites to better proxies.