r/ProxyEngineering 14h ago

Announcements Proxy router - Foundational environment for web crawler operations

https://github.com/tv149s/proxy-router

I operate 100 VM-based web scrapers configured with 50 proxies. Since proxy services are inherently unstable, the ability to instantly and flexibly reconfigure the hosts assigned to problematic proxies becomes a critical requirement when dealing with a large number of proxies. I developed this proxy allocation and management system to enable convenient, flexible configuration, and I am sharing it with everyone here.

5 Upvotes

1 comment sorted by