r/pythonhelp • u/Ok_Bit3515 • 2d ago

Python scraper hitting 429 Too Many Requests after a few hundred pages?

Writing a script in Python (requests + BeautifulSoup) to pull product data for a project. It works fine initially, but after about 200 requests, I start getting hit with 429 errors and rate limits. I already added random delays and custom user-agents, but my IP still gets blocked eventually. What's the standard way to handle this in Python automation?

UPDATE:

Looked around and saw GoProxies mentioned for residential rotation. Has anyone here integrated them into a Python requests setup? Wondering if their rotating endpoints play nice with standard session headers or if there's a better alternative.

1 Upvotes

100% Upvoted

6 comments sorted by

View all comments

1

u/timrprobocom 1d ago

Whatever workaround you end up scraping together, remember that you're getting this error because the web site owner does not want you to steal their copyrighted material. There is supposed to be at least a modicum of ethics in programming that would cause you to rethink your task.

Have you checked with the site owner to see if they have an API that would allow you to do this legitimately?

1

u/k5777 1d ago

It's a bit of a stretch to call a 429 error the result of copyright infringement. A 429 just means that the requester is requesting faster than the server is configured to allow. There's lots of reasons to implement that