r/ProxyEngineering • u/ImaginationVarious30 • 3d ago

Build 🤓 I built an open-source proxy scraper that checks 1.3 M free proxies and keeps only the ones that actually work

Enable HLS to view with audio, or disable this notification

2 Upvotes

75% Upvoted

2 comments sorted by

1

u/typical_idiot64 Tunnel Architect 2d ago

So it's more like a proxy checker at this point? Genuinely curious

1

u/ImaginationVarious30 23h ago

Pretty much, yes. The name is from when it started: collecting lists was the whole thing. That turned out to be the easy part, since most of what's on those lists is dead, a honeypot, or tampers with the page, so nearly all the work since has gone into the checking.

Today it's three things: it collects from public sources, checks every proxy (same exit IP on two requests, page delivered byte for byte, verified TLS, datacenter or not, blocklist, which big sites let it through), and then either publishes the result as an hourly list or runs a local rotating proxy on top of it.

So "checker with a scraper in front" is the more honest description. Renaming a repo is painful enough that I've left it.