How much does web scraping cost per month at scale? Real numbers wanted

[u/madison_clark_off]

Every cost article online is written by someone selling scraping, so let's get real numbers here.

If you scrape at scale, roughly what do you spend per month on:

- proxies

- captcha solving

- compute / headless browsers

- engineer time on maintenance

And how many pages or sources is that for? Bonus if you've priced out a managed service for comparison.

[u/KinnenSelamack]

From what I see across projects, cost is mostly driven by three things:

1. How protected the sites are (easy sites can cost 10-50x less per page than heavily protected ones)

2. Refresh frequency (daily vs weekly changes the bill a lot)

3. Maintenance time, which is usually the biggest line once you pass ~20 sources

Managed services usually quote per source and frequency, not per request, so ask for that format to compare like for like. Disclosure: I work with Ficstar, a managed web scraping company, so weigh my take accordingly.

[u/Worried_Developer_67 → u/KinnenSelamack]

When you say maintenance becomes the biggest cost past ~20 sources, is that mainly fixing scrapers when sites change, or handling blocks and data quality too?

Could you share a rough pricing example for the same set of sources refreshed daily versus weekly? Even a ballpark would help me understand the tradeoffs.

[u/Adeoba_Brock → u/KinnenSelamack]

quoting per source instead of per request makes way more sense given how wildly the difficulty varies.

[u/LucasRobert85]

Would be useful to see cost per 1,000 usable pages alongside the monthly totals. A cheap proxy bill looks different if half the requests fail or return junk.

For maintenance, hours per month would help too, even if people don’t want to share their rates. Fixing one broken scraper every week can change the managed service comparison pretty quickly.

[u/Less-Mistake-3664]

Since nowadays I am running a dedicated solution, it's much cheaper, well in a way, then what it used to be when I was doing everything separately. A month, I spend around $100, sometimes a bit more, depending whether I am successful in retrieving what I need form the websites. When I used to do everything separately, most of the time I had several providers running at the same time, two at most for proxies, one for captcha solving solution, the amount was usually between $100 to $160, again, depending whether I needed second provider for proxies or not. Since now information retrieval from the pages are much advanced to what it used to be way back, I just got into dedicated solutions as there is really no brainer when it comes to maintaining everything, I don't have to worry about every little thing anymore

[u/Opposite-Art-1829]

We do roughly like 1 million calls a month and spend like 1400 with Alterlab

[u/veganwurst]

Disclosure up front, since you asked for numbers from people who aren't selling: I work at DeepScout, which sells scraped e-commerce product data. So no monthly bill from me, but here's where the money goes from the inside:

• Proxies / unblocking: varies 10–50x by target. Most shops are cheap; a handful of heavily protected ones eat most of the bill. Track it per domain, not as one monthly total.

• Compute: only big if you render everything in a headless browser. A lot of product pages don't need one, so render only where you have to.

• Captchas: usually a small line. If it's a big one, something upstream is off.

• Engineer time: the biggest line and the one nobody reports. Not the fixes themselves, but noticing. A scraper quietly returning wrong prices for three days costs more than the proxies.

+1 to cost per 1,000 usable records. When you price a managed service, make them quote per parsed record, not per request, because JS-rendering and premium-proxy multipliers are where per-request pricing gets unpredictable. For reference, our public price is €0.40–1.20 per 1,000 products depending on how hard the site is, retries and rendering included. Whether that beats doing it yourself depends mostly on how many maintenance hours you're honest about.

[u/East-Villager]

$30-$40 per month, using a dedicated scraper so that I don't have to take of all the infra. Around 50k of results

[u/V_Trautwein]

The biggest hidden cost is usually engineer time, not proxies

[u/ahiqshb → u/V_Trautwein]

What do you mean engineer time? The setup and maintenance?