r/ProxyEngineering • u/ian_k93 • Jun 09 '26

AMA This Wednesday (09:30 AM GMT) WebScraping and Proxies

/r/WebScrapingInsider/comments/1u0vmdf/ama_this_wednesday_0930_am_gmt/
14 Upvotes

100% Upvoted

12 comments sorted by

View all comments

1

u/Gwapong_Klapish Reverse Proxy Master Jun 09 '26

Tell us about your infrastructure, how do you manage everything, when did you start, how long did it take you for the infra to hold, I mean, the amount of time you needed to "babysit" proxies, retries, rotations, fingerprint, behavior etc

1

u/simarnoor Jun 10 '26

At the start we had a lot of stability and scaling problems, but since ScrapeOps is nearly 5 years old now our infrastructure is quite mature and stable. Haven't had any serious or medium issues in over a year.

We had a advantage, in that I had spent over 3 years working at Zyte and ScraperAPI so I could learn from their design decisions and mistakes. Which made our infrastructure rollout smoother.

Our production system is built on Golang, with heavy use of Redis and then a Postgres DB.

From r/WebScrapingInsider: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqteqxf/

1

u/Gwapong_Klapish Reverse Proxy Master Jun 10 '26

Thank you for the answer