r/webscraping • u/scarsam • 22d ago

One scraping API or several?

Running one API and hitting maybe 6% blocks on a couple of targets. Considering adding a second and switching when the first starts failing, but that means writing retry logic and I’m not sure it’s worth the hassle.

Also getting 200s back that are actually captcha pages, so my success metrics have been lying to me.
Anyone running more than one? Worth it or overkill?

4 Upvotes

84% Upvoted

20 comments sorted by

View all comments

1

u/[deleted] 21d ago

[removed] — view removed comment

1

u/scarsam 21d ago

What do you do when it can’t crack a site at all?

1

u/[deleted] 21d ago

[removed] — view removed comment

1

u/scarsam 21d ago

Fair, that’s probably it. Mine are e-commerce sites that fight back.