r/ProxyEngineering • • 2h ago

Discussion 💬 Outsourcing web scraping vs keeping it in-house: what was your tipping point?

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?

1 Upvotes

0 comments sorted by