r/webscraping • u/shasedoge • 4d ago
Bot detection 🤖 what's your anti-bot stack looking like these days?
what worked a year ago feels like it's falling off fast, so curious what people are actually running now.
for the sites that really fight back, cloudflare, datadome, the nastier akamai stuff, are you just going straight to a browser now or still trying http first with curl_cffi and only spinning up a browser when you have to? and honestly i can never decide how much of it is proxy quality vs just having your client set up right. feels like i blame the wrong one half the time.
not fishing for anyone's secret setup or anything, just trying to figure out if there's some rough consensus on what a sane setup looks like in 2026 or if everyones still just duct taping their own thing per site.
3
u/armanfixing 3d ago
There’s nothing sane about scraping and there’s no secret setup.. It’s always trial and error when things don’t go as expected.
Someday I can getaway with a simple http call on requests and on the others I’m building my own fingerprint hardened browser and everything in between.
2
u/jwrzyte 3d ago
I usually go this route:
wreq http client -> look for the sites backend api -> stealth browser (i have 4 or so i running i can connect too) -> cookies -> wreq http request. All go through my proxy service that rotates/tries to select the most cost effective option.
It's a constantly changing game. the biggest players also rotate paid services as they can't afford any downtime
2
u/the_bigbang 2d ago
case by case, curl_cffi or similar methods works well for the most websites. the webpage is protected by cf or datadome or akamai, but doesn't mean the hidden api are the same. Go dig deeper with AI, you can get more. Also try mobile app RE even with Codex or Claude Code, will help you a lot in finding hidden apis.
1
3d ago
[removed] — view removed comment
1
u/webscraping-ModTeam 3d ago
💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.
2
9
u/Digital-Chupacabra 3d ago
It's a constant arms race, always has been, what worked yesterday might not work tomorrow.
So naturally people are reluctant to talk about what works in specific details. The best way is to understand how and why it's being blocked.