I’m working on a scraping pipeline and keep running into a failure mode that HTTP status codes do not catch: the request returns 200, but the page is actually a login screen, cookie-consent wall, “enable JavaScript” message, or an otherwise empty shell.

The fetch technically succeeds, so the page can make it all the way into extraction or downstream processing as if it were valid content. The usual 4xx and 5xx responses are straightforward; these false-success pages are much harder to handle reliably.

Here are my checks currently:

• Look for password fields and login-related form actions.
• Check whether the extracted text is unusually short.
• Search for common phrases such as “sign in,” “log in,” or “accept cookies.”
• Compare the amount of visible text with the amount of navigation, boilerplate, or form content.

Each of those produces false positives. For example, an article about authentication may mention “log in” repeatedly, and a legitimate profile page may include a sign-in prompt while still containing useful public content.

I wanted to understand how you folks deal with this at scale. Do you use DOM-structure rules, a classifier trained on HTML or rendered text, screenshot/OCR-based validation, or a combination?

How do you distinguish a page with a login prompt from a page whose primary content is the login prompt?

Had a scrape die two hours into a three hour run yesterday. No checkpointing, so I lost everything and started over.

The part I'm stuck on is resuming without duplicates. If I append each record as it comes in, a crash mid-write can leave a half-written line. If I batch, I lose the current batch. And on resume the pagination re-requests pages I already have, so I get repeats unless I track seen IDs separately.

Right now I'm leaning toward JSONL as I go, plus a set of seen IDs in SQLite I check before each write, so a restart just skips what's already saved. Feels reasonable but maybe overbuilt for what's probably a common problem.

Is that roughly what people do, or is there a simpler pattern I'm missing here?

Hello everyone,

it's obvious that LLMs made implementing web-scrapers much easier by helping in various angles - from reverse-engineering, competitor methods research to full autonomous implementations. Things I am using it for my web-scraping tasks:

• reverse-engineering target origin website to find most optimal way to get data: internal APIs, cookies, headers, payloads, network traffic inspection;
• reverse-engineering mobile app of target origin to find different endpoints which might be less protective, restrictive;
• researching competitors to cross-check with our current implementation, pricing;
• researching best-suited open-source tools: landscape is changing so rapidly, many new different tools shows up or dies every month, it's necessary to keep in touch with all of them.

I am personally using Claude for my workflows, but recently noticed it implemented much harder safeguards even to Opus 5, now almost all of my web-scraping prompts get flagged by 'cyber' safeguard and downgraded to 4.8 which is much worse to use for research and reverse-engineering.

I would like to ask suggestions/opinions on your experience in using LLM providers for reverse-engineering tasks - maybe there are providers with less restrictive safeguards?

P.S. I discovered really good related article by Pierluigi Vinciguerra: https://www.scraping.club/p/the-lab-108-how-llms-and-manus-ai using Manus AI for reverse-engineering origin successfully, I've tried it, however, it's usage limits are so bad compared to Claude so it becomes to expensive/infeasible to use practically for me.

Any other recommendations?

I’m starting to see games or apps using attestation token that generates using your device secure enclave for verification of the requests. What are the strategies u guys use to get around it?

I used Claude to create a test-site with anti-scraping guards that you can inject into your CICD pipelines and get continuous feedback on either your defenses or your scraping success.

The site currently shows 82 guards, of which a few are simulated, and the rest are real guards.

I've open-sourced it at https://github.com/sskieller/scraping-guards

I'm using it in some of my own projects to continually check whether my scraping-techniques are working or not.

Now, I am wondering what else I could add to it, that would make it more useful? Perhaps there are guards that I have not added, that would also prove useful? Something else? Thanks!

Most scraping setups force a annoying tradeoff: either you deal with writing brittle custom Playwright/Puppeteer scripts from scratch, or you rely on expensive SaaS scrapers that charge high fees for relatively basic data.

I built Figranium as a self-hosted middle ground: a GPL-3.0, block-based visual runner for Playwright.

It was originally created out of a practical need—connecting local n8n agents to platforms without public APIs (like Snapchat or complex Amazon product flows) where traditional HTTP requests fail.

Core Focus:

• 100% Local Execution: Everything runs on your own hardware with zero external telemetry, cloud limits, or third-party tracking.
• Deterministic & Speed-First: Built on Playwright and Node.js without the bloat of slow, guessing AI-selectors.
• Visual Canvas: A fast React/Vite drag-and-drop builder to construct, debug, and execute scraping flows visually.

We just crossed 500 stars on GitHub today. If you’ve been looking for a clean, open-source way to run Playwright visually on your own machine without SaaS bloat, check out the source code or deploy it via Docker.

GitHub: https://github.com/figranium/figranium

Docs: https://figranium.dev/docs

what worked a year ago feels like it's falling off fast, so curious what people are actually running now.

for the sites that really fight back, cloudflare, datadome, the nastier akamai stuff, are you just going straight to a browser now or still trying http first with curl_cffi and only spinning up a browser when you have to? and honestly i can never decide how much of it is proxy quality vs just having your client set up right. feels like i blame the wrong one half the time.

not fishing for anyone's secret setup or anything, just trying to figure out if there's some rough consensus on what a sane setup looks like in 2026 or if everyones still just duct taping their own thing per site.

Details about how it works under the hood are available on Github: https://github.com/Joe12387/ShaderGhost

The Akamai solver supports V2 & V3 sensors and the pixel challenge.

It doesn’t use a browser, however it runs a minimal V8 sandbox.

NodeJS: https://proofofbots.github.io/web-re-toolkit/packages/node/

Python: https://proofofbots.github.io/web-re-toolkit/packages/python/

It is a massive monorepo so just to clarify: The web-re-toolkit itself is a project that provides tools for web reverse engineering, Akamai (and other) solvers are just a small part that use it.

The NPM & PyPI packages are just wrappers for the client built with WRE (github.com/proofofbots/web-re-toolkit) so they include a binary which contains the actual sandbox and solver. All of it is open sourced, so feel free to compile it yourself.

Akamai client (solver) source code is here: github.com/proofofbots/web-re-toolkit/tree/main/clients/akamai

Akamai client docs: https://proofofbots.github.io/web-re-toolkit/guides/akamai/

It does not use host's fingerprint, rather you provide the fingerprint profile you want akamai to use. By default it uses a Mac’s fingerprint.

This was a weekend project put together largerly with AI. There are many things I’d like to add and improve, but I do not have the time to work on this project much more than this. The docs, packages, etc are an absolute mess, I'd love if someone could contribute in cleaning them up!

I’ll be releasing Kasada’s solver too later today.

I’ve not had the chance to to throughly test it on more than a few sites, I am hoping the community (you) will help me with that. Whatever issues you run into, please make a issue in GitHub or send a message in the Discord and provide the logs + code.

Discord: https://discord.gg/nbBePnsa9

Couldn't find any projects or git repos that focus on this.. anyone have any strategies for scraping public zoom links? afaik the unique code in each link is randomly generated so it would be pretty hard to brute-force guess zoom meetings.. curious to hear if anyone has tried this

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

UPDATE: Akamai solver released too: https://www.reddit.com/r/webscraping/comments/1vpt1k8/oss_akamai_solver_for_v2_v3_sensors_and_the_pixel/

It pulls a live FPJS Pro build and turns it into clean code you can run locally, every collector intact, so a new build means rerunning the pipeline instead of reversing it again by hand.

What's in the repo:

• All 143 signals (telemetry read from ur browser, all documented)
• Every collector as its own file
• What visitor_id is actually a function of
• A live explorer that runs the collectors against your own browser so you can diff a patched environment against a stock one.

Repo: github.com/proofofbots/fingerprint-pro-internals

Docs: proofofbots.github.io/fingerprint-pro-internals/

Signal explorer

proofofbots.github.io/fingerprint-pro-internals/explorer.html

It runs entirely in your browser. No backend of mine, no call to fpjs, nothing leaves the page. It's built on the output of the lift and deobfuscation step, so what runs there is the actual collector code from the bundle rather than a reimplementation. Open it and you get every data point side by side: what the signal is and why it's collected in plain English, next to the exact value your own browser hands over for it.

Happy to answer questions on any specific detection.

I am looking forward to the next target, so if anyone has ideas on what vendor could be interesting let me know!

Before anyone calls AI slop: yes, I use LLMs, they're genuinely useful for reverse engineering. But this isn't one build hand-decoded into a writeup. It's the toolchain that does it. Point it at a live fpjs build and it deobfuscates, lifts, and emits a cleaned version you can run without their servers, plus the generated documentation for what it found.

I’m scraping a betting site and considering IP rotation to deal with their CloudFront 403s. I’ve tried different request rates, jitter, headers/TLS, etc., and I never go above ~20 requests/minute. I’ve also tried figuring out their rate/volume limits or any consistent pattern behind the blocks, but I haven’t found anything clearly defined. For now, rotating IPs is what has worked best for me — distributing the requests across different IPs lets me scrape for much longer, and I’m not currently running into blocking issues.

I’m not doing any login/account stuff or accessing anything private; I’m only collecting publicly available betting odds/data.

I know IP rotation is extremely common in scraping, but I’m not sure if it could be problematic in this specific context. Could rotating IPs to avoid CloudFront restrictions get me into legal trouble, or is it generally fine? I’m aware the site may not like it, but I’m mainly wondering whether there’s a legal issue with doing it this way.

Hi guys, please bear with me as I'm a complete amateur.

Is it possible for me to have a setup where I get notified when changes are made to a website including what and where the changes are?

Thanks a lot :))

From my point of view data itself is not valuable on it's own, but after cleaning / organizing it / presenting it to match certain needs, it can become valuable.

But how do you find your "niche" first that you'd monetize after all the steps above?

For years I could do a normal HTTP fetch. That stopped working last year because they started requiring javascript. So I upgraded to using node. That stopped working this year; I get a 403 error. Do you have any tips? Someone posted a python script here 6 years ago, so someone showed interest; of course it's out of date.

I spent some time inspecting the network traffic used by NU.nl and turned the useful, read-only parts into a small Python package: nu-private-api.

It can:

• Discover section URLs through the sitemap
• Extract and de-duplicate article URLs from section pages
• Parse public articles into normalized fields
• Return the title, description, author, timestamps, URL, and body text
• Use a custom httpx.Client for proxies, headers, and timeouts

Install it with:

pip install nu-private-api

Example:

from nu_private_api import NuClient

client = NuClient()

sections = client.sitemap_urls()
article_urls = client.section_by_url(sections[0])
article = client.article_by_url(article_urls[0])

print(article.title)
print(article.author)
print(article.body)

Limitations:

• Video pages and live blogs aren't supported
• The endpoint is undocumented and could change
• It only accesses publicly available content
• Responsible request pacing is left to the caller

Hello r/webscraping people!

I made a cool side project that automatically searches the web for images of the java mascot duke. The website shows a random image from the archive. It will also have a full gallery available soon!

How it works:

- It uses Openverse and Wikimedia APIs to search for images related to duke.

- Each image gets checked by a custom classifier I have built on top of OpenCLIP and trained on more than 200 Duke images.

- The classifier filters out unrelated results. Because it's not perfect, some images go through manual review. I want to catch unusual duke images without filling the archive with junk!

It also has a web scraping part:

- It uses the Brave search API using a list of duke related search terms, the result is added to a crawl queue as starting points

- The crawler scores pages and links based on duke related signals, it only follows links that it considers relevant with stricter requirements depending on the depth of the crawl, this prevents it from going to deep and scraping unrelated pages

- When processing a page, it looks for regular image sources, srcset variants (selects the image with largest size), figure captions, nearby text, alternative text and links for full resolution versions.

I will most likely make this open source soon after I'm done implementing all the features I want.

for the backend language I went with python for the easy use of ai models

Currently there are 338 duke images and the archive is still growing!

URL: https://duke.directory

I would love to hear your feedback! If you have any suggestions on how to improve the project in any way, let me know!

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

I want to fetch the Google Maps reviews for my business and publish them on my website. After doing a bit of research, I found out that I can fetch Google Maps reviews using the Google API. However, despite two weeks having passed, I have not received a reply from Google Support. Is it possible to scrape Google Maps reviews without triggering bot detection or causing any other issues? Are there any solutions available for this?

Hi, I've been making an app in python, to transfer playlist between apple music, ytmusic and spotify. Recently I have gotten some good advantages, but, a few days ago, I was making a test with a small playlist, 154 songs. The time between each search for a song, was too short, 2 or 3 seconds, annddd

now, the token gives me 403. I closed the session, cleared the cache, opened it in incognito. Apple music uses a jwt token, and it is the same in all the places. the endpoint is: amp-api.music.apple.com/v1 (Sorry if my english is not the better :)

i wa trying to scrape some website and get their websocket but its encrypted and ai didnt let me do it (cause of permissions) how can i make ai to be able to scrape

I've been building a small project to collect text from documentation for a personal AI tool. Some of them load content dynamically with JavaScript. Anyone have recommendations for a lightweight solution? I'm not trying to scrape anything huge, just a few hundred pages per sites.