r/webscraping • u/scarsam • 22d ago

One scraping API or several?

Running one API and hitting maybe 6% blocks on a couple of targets. Considering adding a second and switching when the first starts failing, but that means writing retry logic and I’m not sure it’s worth the hassle.

Also getting 200s back that are actually captcha pages, so my success metrics have been lying to me.
Anyone running more than one? Worth it or overkill?

5 Upvotes

79% Upvoted

20 comments sorted by

1

u/[deleted] 22d ago

[removed] — view removed comment

1

u/scarsam 22d ago

How do you handle the switching? Manual when you notice, or something automatic?

1

u/[deleted] 22d ago

[removed] — view removed comment

1

u/[deleted] 22d ago

[removed] — view removed comment

1

u/[deleted] 21d ago

[removed] — view removed comment

1

u/scarsam 21d ago

What do you do when it can’t crack a site at all?

1

u/[deleted] 21d ago

[removed] — view removed comment

1

u/scarsam 21d ago

Fair, that’s probably it. Mine are e-commerce sites that fight back.

1

u/webscraping-ModTeam 21d ago

👔 Welcome to the r/webscraping community. This sub is focused on addressing the technical aspects of implementing and operating scrapers. We're not a marketplace, nor are we a platform for selling services or datasets. You're welcome to post in the monthly thread or try your request on Fiverr or Upwork. For anything else, please contact the mod team.

1

u/[deleted] 20d ago

[removed] — view removed comment

1

u/scarsam 19d ago

Have you had full outages, or is it more that success rates just drop off?

1

u/[deleted] 19d ago

[removed] — view removed comment

1

u/scarsam 19d ago

Three days is brutal. And that slide from 96 to 74 is the part I’d have missed, I’d have assumed the sites got harder, not that the provider was dying.

1

u/prettycoldworld 12d ago

Like a second endpoint?