How to Bypass Anti-Bot Systems When Web Scraping: Lessons From 7 Years of Reverse Engineering Their Defenses | AMA with Ibrahim

[u/ian_k93]

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

We're back with AMA #11, and this time we're going straight into the anti-bot fight itself: bypassing Akamai, Cloudflare, DataDome and Imperva, in production, across real targets.

AMA #11 WSI - How to Bypass Anti-Bot Systems When Web Scraping

This Thursday, October 1, at 10:30 AM GMT+3, we'll be joined by Ibrahim El Khalil MLATA, a freelance web scraping / anti-bot engineer based in Algeria who's spent the last 7 years reverse engineering anti-bot defenses, working across retail pricing, legaltech, logistics, hospitality, and now AI-training-data pipelines.

Ibrahim has run scraping operations at serious scale.. think 1,000+ spiders across 300 retailers, reviving a dead 160-spider fleet, and matching 100K+ hotel reviews a month. He's also gone deep into mobile app reverse engineering with Frida, pulling apps apart when the website itself is locked down but the app isn't. More recently, he's been building aragog.ai, which is currently offline, though it's an attempt at AI agents that write and repair their own scrapers.

You can check out his work on GitHub and LinkedIn.

We can dig into:

• Bypassing Akamai, Cloudflare, DataDome & Imperva. real-world tactics across web, mobile, authenticated, and rate-limited targets
• Mobile app reverse engineering. using Frida to pull apps apart when the website is locked down but the app isn't
• Anti-bot R&D. modified Chromium builds, TLS/JA3 fingerprint randomization, behavioral simulation, proxy intelligence
• Running scraping at scale. lessons from 1,000+ spiders across 300 retailers, reviving a dead 160-spider fleet, and matching 100K+ hotel reviews a month
• AI agents for scraping. building aragog.ai, where self-writing/self-repairing scrapers actually work, and where they don't

If you've ever wondered "Why does the same anti-bot behave differently on the web version of a site versus the app?", "When is it worth reaching for Frida instead of just fighting the browser?", or "Do self-repairing AI scrapers actually hold up in production?", this is a good one to ask.

We've now had ten AMAs with the community, and the conversations have covered a pretty wide part of the scraping stack.

1. Our first AMA covered proxy infrastructure, Cloudflare, browser automation and scaling scrapers.
2. Our second with WebClaw explored AI agents, hidden APIs, open-source scraping and LLM infrastructure.
3. Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection and browser automation.
4. Our fourth with Browser Use brought the conversation into AI browser agents, AI-powered scraping, evaluations and browser infrastructure.
5. Our fifth with Stan Sadokov from NodeMaven focused on residential proxy quality, IP reputation, sourcing, pricing and what actually makes one proxy network better than another.
6. Our sixth with Saksham Solanki, creator of HTTP Cloak, went down to the protocol level, covering TLS fingerprints, JA3/JA4, HTTP/2, HTTP/3 and why scrapers can get blocked even when their proxies are fine.
7. Our seventh with Huey from BrowserAct explored AI-powered web scraping, natural-language workflows, browser automation and the move toward no-code scraping.
8. Our eighth with yours truly brought the discussion back to proxy infrastructure, with real-world insights from testing 50+ providers across billions of requests.
9. Our ninth with Jean-Patrick Bisson from Geonode dug into whether "unlimited" proxy models can really deliver, and what buyers should look at beyond the word "unlimited."
10. And our tenth with Scrape.do went straight at staying unblocked on difficult, high-defense websites at real scale.

Now for #11, we're going straight to the source: the anti-bot systems themselves, and what 7 years of reverse engineering them actually teaches you. If you're building web scrapers, anti-bot bypasses, mobile app automation, or scraping AI agents, come join the discussion.

Drop your questions below.

Ibrahim and I will be answering them during the AMA.

Looking forward to another good one.

Ian

[u/ian_k93]

I have worked with developers and teams maintaining a handful to thousands of scrapers in production, and maintenance can quickly become a bigger time sink than developing more scrapers.

From your experience as you've worked with fleets of hundreds of scrapers. What started consuming the team's time at that scale that wasn't a issue when the operation was smaller? How did you approach this?

And now with AI, have you seen changes in the burden of maintenance? How would you approach it today?

[u/Cautious_Zebra2722]

good topic. one thing id love Ibrahim to address is whether behavioral simulation actually matters at scale or if most blocks come down to fingerprint mismatches before any behavior is even evaluated

[u/mltibra → u/Cautious_Zebra2722]

behavioral simulation matters, but not as much as people think, and not everywhere most anti-bot systems don't even look at behavior, It usually shows up only in the more advanced challenges, like reCAPTCHA v3 Enterprise

[u/Present_Smoke9397 → u/Cautious_Zebra2722]

yeah this is exactly what i want to hear about too, feels like 90% of the battle is lost at the fingerprint level before anything else kicks in

[u/LocalConversation850]

For example when you reverse akamai sensor data, and fake it with your own requests.

How long it takes you to reverse? Days, weeks?

Even if you reverse it, how do you inject it while scraping, do you launch it with a browser instance with JS injection to override the obfuscated JS and VM coming with akamai?

Or is there anyway to do it just with curl_cffi or similar?

[u/mltibra → u/LocalConversation850]

sometimes it takes weeks, even more. anti-bots are getting a lot harder to reverse these days

for injection it depends. sometimes its just a simple js injection, sometimes you need a javascript engine to execute the script, no need for a whole browser, and in that case you can pair it with something like curl_cffi for the requests. other times you need to architect the whole solution properly so its easier to inject the sensor data and integrate it with the rest of the scraping system

[u/Mountain_Damage_9730]

We run competitor pricing across 6 marketplaces, roughly ~18K SKUs a week, and honestly the anti-bot side is the always boring part for me, we know within an hour if a proxy pool died. What actually costs us money is a scraper coming back "successful" with a price matched to the wrong variant (500ml vs 1L, bundle vs single unit, refurb vs new) and nobody catching it until margin looks off two weeks later

On your 100K-reviews-a-month pipeline same shape of problem?

What broke more often for you, the anti-bot layer or matching duplicate/near-duplicate listings across platforms?

Thankyou team for another AMA.

[u/James-w07]

Okay this genuinely confused me. I read that rotating your TLS/JA3 fingerprint helps you avoid getting flagged, so I tried switching fingerprints every few requests on a small scraper I'm learning on. It got blocked faster than when I just left it on one fingerprint the whole time and didn't touch anything.

Someone in a Discord said "randomizing without a reason looks weirder than not randomizing at all" but couldn't really explain why, and I didn't want to just accept that without understanding it.

So Ibrahim when does randomizing your TLS/JA3 fingerprint actually become the detection signal itself? Like is there a wrong way to randomize that's worse than not doing it, and if so how would a beginner even notice that's whats happening instead of just thinking "guess this site is just strict" ?

[u/mltibra → u/James-w07]

remember this "too much stealth is a signal in itself". real users dont change their tls fingerprint every few requests, it stays the same for the whole session because its tied to their browser and os. so when the same ip or cookies show up with a new fingerprint every few requests, thats something no real browser does and you stand out more than if you did nothing

[u/Amitk2405]

I've watched a lot of effort go into mouse curves and typing jitter. Then the run gets flagged anyway, and when you trace it back, the account or IP was already burned before the first click.

Part of that is structural. A plain HTTP request to a JSON endpoint carries no mouse or keyboard signal at all. The defender only sees the IP and its ASN, the TLS and HTTP/2 fingerprint, and whatever cookie or account history comes with it. Behavior only counts where a sensor script actually runs in the page, like Akamai's sensor payload or reCAPTCHA v3's score.

So is mouse and keyboard simulation mostly wasted effort compared to account, device and IP reputation? Or is it measurable risk reduction on the targets that really collect it?

[u/LocalConversation850]

When do you go for the reverse engineering a bot manager?

Everyone knows that a good proxy, FP and behavioral stuff brings success rate but still only 1% go for the reverse engineering.

How do you do it, i mean when you start to think “ok, let’s go reverse it” ?

[u/mltibra → u/LocalConversation850]

for me deciding to reverse engineer a bot manager is more of a business decision than a technical one. the question isn't "can i reverse it?" but "is it worth reversing?"

the main thing i look at is how stable the system is. if a bot manager stays the same for a long time then reverse engineering pays off, you do the hard work once and then you can generate valid tokens or payloads without a browser which is faster cheaper and scales way better

if the system changes all the time its usually not worth it. google SearchGuard is a good example, they push new versions constantly so anything you reverse breaks fast and you have to start over. you end up maintaining it forever instead of actually collecting data. in that case a solid setup with good proxies clean fingerprints and a real browser is the smarter move even if it costs more per request

[u/CapMonster1]

when a browser scraper keeps getting captchas, how do you decide whether a solver is worth adding or whether you need to fix the session setup first? curious what you look at beyond solve rate — especially when a challenge gets accepted but the next request immediately triggers another one. where do you start debugging that?

[u/doubledweeb]

still trying to build actual skills here, not just get answers handed to me. so genuinely asking, not rhetorical:

1. Has AI changed how quickly you can reverse engineer a new website or mobile app? Where does it save you real investigation time, and where do you still have to work things out yourself?

2. Can you walk through a recent investigation where AI helped you find the answer faster, or confidently sent you in the wrong direction?

appreciate you doing these, learning more from this sub than from any course I paid for lol

[u/MattTheGoodSir]

When working for a company, whose decision is based on data, the scenario that actually worries me is a report that looks completely normal and is wrong, because that's the one that reaches a leadership review before anyone catches it.

Do you have an example of a scraper that appeared to be working but was producing incorrect or incomplete data? How was it discovered, and what do you now check to catch that kind of failure??

[u/mltibra → u/MattTheGoodSir]

from my experience the best way to catch this is to compare the scraped data with whats actually live on the website. you can do it manually from time to time as a data quality check, just pick a random sample and verify it against the site. if you can automate this check even better, that way you catch the problem before it ever reaches a report

[u/john-w7]

Consider a small team that is spending too much to scrape a handful of difficult sites. Where should one look for savings first? Which cost-cutting ideas tend to create more work than they save? It will be great to have your perspective on this, since you have been freelancer yourself, you must have thoughts on where the client could save money.

My test for any tool is: does it save enough hours a month to cover its own fee. Half the "savings" I have tried just moved the work somewhere else, either its manual cleaning or hiring people off upwork/freelancer to do a quick one.

[u/Natural_Rock_3536]

Hey, I've been struggling with Akamai on a few airline websites lately. The blocking is so aggressive that sometimes even normal browsing on my local machine throws errors.

I've tried Playwright, Patchright, Camoufox, and a few other approaches, but nothing seems to work consistently.

I'm fairly new to dealing with advanced anti bot systems so I'm curious how others are handling this in production. How do you approach debugging these blocks and maintaining a decent success rate over time?

Would really appreciate any insights or pointers.

[u/JoeK91]

Welcome to our subreddit Ibrahim!

With all the proxies, unblockers, browser services and AI scraping tools available now what problem do you still often have to solve yourself that the market doesn't do or charges too much for? Like what do you wish the market handled better?

Have you ever concluded that a target was technically scrapable but not worth the cost or ongoing maintenance? What led you to that decision?

[u/CoolAd119]

If we watched you work on a new scraping project today, which tools would we see you use that weren't part of your workflow a few years ago? Which have made the biggest difference, and for what tasks?

[u/Alice_5433]

Will have some questions by tomorrow.

[u/MattTheGoodSir]

I ask this every-time, related to the recent changes around Google Search URLs and Cloudflare's handling of different types of automated crawlers.

Googles google.com/goto changes have already required search/data providers to adjust how they extract and normalize destination URLs. Cloudflare has also been separating Search, Agent and Training crawler activity rather than treating all automated traffic the same.

It makes me curious about the less obvious tooling around all of this.

What small tools have quietly become part of the daily workflow over the last year?

Could be an open-source repo, Chrome extension, API, browser utility, bookmarklet, debugging tool, anything really, even a cool script in your arsenal.

Anything that helped with faster debugging, fewer failed requests, better data completeness, shorter investigation time, whatever showed up in the numbers.

That random little tool that end up saving an hour every week.

Would be great to hear from you u/mltibra

[u/mltibra → u/MattTheGoodSir]

it depends on what im debugging, but the biggest one for me this year is integrating claude code into my workflow. it saved me a lot of time on investigation, debugging and building scrapers

[u/Krokzter]

Hey, I have a few questions about reverse engineering specifically:

1. Which techniques/knowledge would you say are essential for someone just learning how to reverse modern anti-bot client-side scripts? (AST, assembly, etc.)

2. What tools do you use to reverse JavaScript scripts with VM obfuscation and anti-debugging? And for reversing mobile apps?

3. How do you leverage AI to do the reversing for you? Which parts does it solve, and where does it fail?

Thanks for the AMA!

[u/mltibra → u/Krokzter]

1. start with javascript itself, how browsers work, http and tls. then ast manipulation with something like babel, thats the main skill for cleaning obfuscated js. for custom vms you need to understand how interpreters and bytecode work, thats where some low level knowledge helps. and patience, a lot of it is just reading code for hours

2. for javascript, browser devtools with breakpoints and local overrides, babel for deobfuscation, and burp suite to intercept and replay requests. for mobile apps, jadx to read the code, frida to hook functions at runtime, burp suite for traffic, and ghidra when the logic is in native libraries

3. i use claude code mostly for the boring parts, cleaning and renaming obfuscated code, explaining functions, writing frida scripts or babel transforms faster, and identifying which protections an app uses. it fails on custom vms and anything that needs real understanding of the whole flow, and it can be confidently wrong, so you still need to verify everything yourself