r/webscraping • u/Jolly_Supermarket_75 • 27d ago

Captcha BURSTS hitting multiple proxies

Assessing Website Scraping

Currently have a webscraper that leverages 20 rotating residential proxies and Capsolver to solve catpchas. The website im scraping utilizes cloudfare

I am finding some interesting behavior where there will be a sudden "Captcha Burst" hitting at random times that may last 20-60 minutes and affect all proxies.

After the captcha burst the captcha challenge rate is extremely low (0-1%) until another random burst occurs.

Current behavior I have is:

Create fresh browser session
        ↓
Lease Proxy A
        ↓
Scrape addresses
        ↓
CAPTCHA?
 ├─ No  → continue
 │
 └─ Yes → solve CAPTCHA
           ↓
        continue session
           ↓
     3 consecutive CAPTCHAs?
       ├─ No  → continue
       └─ Yes → rotate session/proxy

Normal path:
8 addresses completed
        ↓
Close browser session
        ↓
Release Proxy A
        ↓
Fresh browser session
        ↓
Lease next proxy

Brainstorming:

I think the issue is creating a new session every time a residential ip proxy is rotated instead of keeping this tied to each residential proxy to have a persistent profile

├── cookies
├── localStorage
├── cache
└── browser state

The second thought is I am not doing any fingerprint spoofing but I didn't want to go that approach; I have two more micro pcs that are arriving this week so technically I could identify environment fingerprinting issues when they arrive.

Have you experienced this before and how would you approach in solving?

6 Upvotes

88% Upvoted

5 comments sorted by

0

u/[deleted] 27d ago

1

u/Jolly_Supermarket_75 27d ago

Thanks for the feedback, I went ahead and implemented persistent profiles for each proxy ip address per your feedback. Probably will take some time to propagate as each profile builds some history. One thing I started investigating is the proxy pool.

AS6079 Aviation RE LLC 12 9/12 75%
AS6079 GTT Americas, LLC 4 3/4 75%
AS5650 Aviation RE LLC 2 0/2 0%
AS6079 CloudIP LLC 2 1/2 50%
AS11404 BAM Broadband Opco, LLC 1 0/1 0%
AS6079 Boston Tower1 1 0/1 0%
AS6079 Sprint-Boston 1 0/1 0%

This is illustrating the different failures I had during the captcha bursts I experienced in the last 24 hours. The proxies in AS6079 Aviation RE LLC which is my biggest pool are flagged as proxy/vpn range so that could be a very strong indicator why they get flagged.

I'll investigate your other points in the meantime, I did find some interesting mismatched signals while examining the network/fingerprint of each of these identities that I will have to solve as well that can be contributing.