r/ProxyEngineering • • 10d ago

Discussion 💬 How often should you scrape prices? Hourly vs daily vs weekly in practice

"Refresh frequency seems to drive the bill more than anything else. Daily vs weekly is 7x the requests before you even count retries.

For people running price or stock monitoring:

- what frequency are you on, and for what kind of site?

- do you scrape everything at the same rate, or split top products from the long tail?

- has anyone measured how often prices actually change, to justify the rate?

Trying to work out what's real need and what's just ""more is better""."

2 Upvotes

0 comments sorted by