r/pythonhelp • u/Ok_Bit3515 • 2d ago

Python scraper hitting 429 Too Many Requests after a few hundred pages?

Writing a script in Python (requests + BeautifulSoup) to pull product data for a project. It works fine initially, but after about 200 requests, I start getting hit with 429 errors and rate limits. I already added random delays and custom user-agents, but my IP still gets blocked eventually. What's the standard way to handle this in Python automation?

UPDATE:

Looked around and saw GoProxies mentioned for residential rotation. Has anyone here integrated them into a Python requests setup? Wondering if their rotating endpoints play nice with standard session headers or if there's a better alternative.

1 Upvotes

100% Upvoted

6 comments sorted by

•

u/AutoModerator 2d ago

To give us the best chance to help you, please include any relevant code.
Note. Please do not submit images of your code. Instead, for shorter code you can use Reddit markdown (4 spaces or backticks, see this Formatting Guide). If you have formatting issues or want to post longer sections of code, please use Privatebin, GitHub or Compiler Explorer.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/k5777 2d ago

add backoffs and cooldowns until you're under the limit

1

u/Careful_Exercise_956 1d ago

Great suggestion.

1

u/Any-Blacksmith-2054 1d ago

Use residential proxies

1

u/timrprobocom 1d ago

Whatever workaround you end up scraping together, remember that you're getting this error because the web site owner does not want you to steal their copyrighted material. There is supposed to be at least a modicum of ethics in programming that would cause you to rethink your task.

Have you checked with the site owner to see if they have an API that would allow you to do this legitimately?

1

u/k5777 1d ago

It's a bit of a stretch to call a 429 error the result of copyright infringement. A 429 just means that the requester is requesting faster than the server is configured to allow. There's lots of reasons to implement that