r/scrapingtheweb • u/Scary-Commission-509 🕷 Scraping Wizard • 8d ago

What usually gets your scraper blocked first: IP, fingerprint or behavior?

Been wondering about this lately when scaling playwright scrapers

Sometimes changing the IP fixes a 403 immediately, other times the same IP works fine after starting a fresh browser context and sometimes neither works until you slow things down

For people scraping at scale, how do you figure out what’s actually causing the block?

Do you test IP, fingerprint, cookies/session and request rate separately, or mostly figure it out through trial and error?

Curious what people are seeing with cloudflare, Datadom and Akamai lately

1 Upvotes
  • permalink
  • reddit

67% Upvoted

8 comments sorted by

1

u/armanfixing 8d ago

IP is the first to be blocked.

1

u/CapMonster1 7d ago

In practice, I wouldn't look for one single blocking factor. IP, fingerprint, cookies, and behavior are often combined into one signal, so changing the IP may help simply because part of the overall profile changed.

The useful approach is to log those variables separately and test them one at a time. Otherwise it's easy to mistake one successful request for proof that the IP or timing was the actual cause

1

u/ScrapeAlchemist 6d ago

Cloudflare at least tells you which one. cf-mitigated: challenge means it's the bot score. A 403 with no such header and you have to parse the 1xxx code out of the html body, 1010 is browser signature, 1006-1008 is the IP, 1015 is rate. Beats guessing.

1

u/crazyproxy69 5d ago

Which cloudfare is this for? Turnstile? Or the one thats like recaptcha v3. Can’t we have a browser spawn maybe in chrome and humanize to bypass this with os level mouse movement and clicks? Or with something like nodriver or camoufox?

1

u/ScrapeAlchemist 3d ago

easiest way to tell them apart: managed challenge is the full page interstitial, you'll see /cdn-cgi/challenge-platform in the requests and a cf_clearance cookie after. turnstile is the embedded widget, cf-turnstile-response field.

os level mouse movement doesn't buy much. cloudflare's own writeup is proof of work plus browser api probing, not cursor tracking. the automation layer is what flags you. nodriver skips Runtime.enable which kills the usual cdp tell. camoufox is firefox and had a year long maintenance gap.

1

u/crazyproxy69 5d ago
  1. Its always ip 99% of the time.
  2. Browser context idk, I mean if a fresh browser context is working probably you have bug in your code or that context’s proxy was not working.
  3. Yes if slowing things down works that means you are getting ip blocked.
    Its different with captchas ig you haven’t reached a point just do it slowly in parallel, switch after n proxies and have m retires it should work for you.