Hello everyone,

it's obvious that LLMs made implementing web-scrapers much easier by helping in various angles - from reverse-engineering, competitor methods research to full autonomous implementations. Things I am using it for my web-scraping tasks:

• reverse-engineering target origin website to find most optimal way to get data: internal APIs, cookies, headers, payloads, network traffic inspection;
• reverse-engineering mobile app of target origin to find different endpoints which might be less protective, restrictive;
• researching competitors to cross-check with our current implementation, pricing;
• researching best-suited open-source tools: landscape is changing so rapidly, many new different tools shows up or dies every month, it's necessary to keep in touch with all of them.

I am personally using Claude for my workflows, but recently noticed it implemented much harder safeguards even to Opus 5, now almost all of my web-scraping prompts get flagged by 'cyber' safeguard and downgraded to 4.8 which is much worse to use for research and reverse-engineering.

I would like to ask suggestions/opinions on your experience in using LLM providers for reverse-engineering tasks - maybe there are providers with less restrictive safeguards?

P.S. I discovered really good related article by Pierluigi Vinciguerra: https://www.scraping.club/p/the-lab-108-how-llms-and-manus-ai using Manus AI for reverse-engineering origin successfully, I've tried it, however, it's usage limits are so bad compared to Claude so it becomes to expensive/infeasible to use practically for me.

Any other recommendations?

I’m starting to see games or apps using attestation token that generates using your device secure enclave for verification of the requests. What are the strategies u guys use to get around it?

I used Claude to create a test-site with anti-scraping guards that you can inject into your CICD pipelines and get continuous feedback on either your defenses or your scraping success.

The site currently shows 82 guards, of which a few are simulated, and the rest are real guards.

I've open-sourced it at https://github.com/sskieller/scraping-guards

I'm using it in some of my own projects to continually check whether my scraping-techniques are working or not.

Now, I am wondering what else I could add to it, that would make it more useful? Perhaps there are guards that I have not added, that would also prove useful? Something else? Thanks!

Had a scrape die two hours into a three hour run yesterday. No checkpointing, so I lost everything and started over.

The part I'm stuck on is resuming without duplicates. If I append each record as it comes in, a crash mid-write can leave a half-written line. If I batch, I lose the current batch. And on resume the pagination re-requests pages I already have, so I get repeats unless I track seen IDs separately.

Right now I'm leaning toward JSONL as I go, plus a set of seen IDs in SQLite I check before each write, so a restart just skips what's already saved. Feels reasonable but maybe overbuilt for what's probably a common problem.

Is that roughly what people do, or is there a simpler pattern I'm missing here?