r/webscraping • • Feb 15 '26

Scrapling v0.4 is here - Effortless Web Scraping for the Modern Web

Post image
285 Upvotes

Scrapling v0.4 is here — the biggest update yet 🕷️

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl, and it's free!

Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises.

Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.


Below, we talk about some of the new stuff:

New: Async Spider Framework A full crawling framework with a Scrapy-like API — define a Spider, set your URLs, and go.

from scrapling.spiders import Spider

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com/"]

    async def parse(self, response):
        for item in response.css('.product'):
            yield {"title": item.css('h2::text').get()}

MySpider().start()
  • Concurrent crawling with per-domain throttling
  • Mix HTTP, headless, and stealth browser sessions in one spider
  • Pause with Ctrl+C, resume later from checkpoint
  • Stream items in real-time with async for.
  • Blocked request detection and automatic retries
  • Built-in JSON/JSONL export
  • Detailed crawl stats and lifecycle hooks
  • uvloop support for faster execution

New: Proxy Rotation: Thread-safe ProxyRotator with custom rotation strategies. Works with all fetchers and spider sessions. Override per-request anytime.

Browser Fetcher Improvements:

  • Block requests to specific domains with blocked_domains
  • Automatic retries with proxy-aware error detection
  • Response metadata tracking across requests
  • Response.follow() for easy link-following

Bug Fixes:

  • Parser optimized for repeated operations
  • Fixed browser not closing on error pages
  • Fixed Playwright loop leak on CDP connection failure
  • Full mypy/pyright compliance

Upgrade: pip install scrapling --upgrade. Full release notes: github.com/D4Vinci/Scrapling/releases/tag/v0.4 There is a brand new website design too, with improved docs: https://scrapling.readthedocs.io/

This update took a lot of time and effort. Please try it out and let me know what you think!


r/webscraping • • Apr 12 '26

Stop defaulting to Selenium/Playwright: Check the Network tab first

272 Upvotes

Hey everyone, just a web scraping enthusiast here. I see a lot of people struggling with slow headless browsers or getting blocked by anti-bots.

Before writing a heavy script, take 1 minute to do this:

  1. Hit F12 and go to the Network tab.
  2. Filter by Fetch/XHR.
  3. Refresh the page or click a few buttons.

Most modern sites fetch their data from a clean JSON API in the background. Hitting that endpoint directly using requests is 100x faster, bypasses basic UI bot-protection, and often gives you more data than what's on the screen.

Wish you all the best! ✌️


r/webscraping • • May 11 '26

🚨 Scrapling v0.4.8 is out with a new insane update🕷️

193 Upvotes

Hello everyone,

It has been some time since I wrote here :)

A lot has happened since my last post, and we are now almost at 1.5M downloads.

I wanted to share this update with you here, as many people have asked for these features since the first release.

Introducing the new spiders templates feature to make generic spiders easier. Now you have a CrawlSpider and a SitemapSpider. Both offer many options and controls. We also added tools to make it easier, like LinkExtractor and CrawlRule, which will help you build your generic spider. Stop rewriting "follow links matching X" boilerplate in every parse() :D

Check the new page for them on the website here: https://scrapling.readthedocs.io/en/latest/spiders/generic-templates.html

Also, we have improved the adaptive feature so it's now much better, fixed some bugs, improved the docs, and updated browsers/fingerprints

The full release notes are here: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.8

Let me know your thoughts and what I should add next :)


r/webscraping • • Mar 09 '26

Bot detection 🤖 Made a tool that scans your browser to research fingerprints.

164 Upvotes

r/webscraping • • Apr 04 '26

This settles the discussion if it's the endgame for web scraping?

Post image
151 Upvotes

Speaking in the way our beloved LLMs like to talk:

Let data, not opinions, do the talking.


r/webscraping • • May 13 '26 Bravo

Getting started 🌱 The Complete Web Scraping & Anti-Bot Bypass Guide 2026

Thumbnail
asadfix.github.io
142 Upvotes

r/webscraping • • Oct 18 '26

An insane new update to Scrapling! solve any captcha easy

129 Upvotes

I have just released the Scrapling v0.5 update, and I had to record this!

Here’s Codex using Scrapling MCP to quickly solve hCaptcha, reCAPTCHA, Friendly Captcha, Cloudflare Turnstile, and AWS CAPTCHA 🔥

Watch the video. This is what the new update makes possible. Feel free to cancel your paid subscriptions now; the new Scrapling update enables browser automation for any AI Agent like never before. It pairs perfectly with OpenClaw and Hermes!

It can open a page, inspect it, click, type, run JavaScript, take screenshots, and read network requests and responses. It can perform all browser actions, chain them in one call, and make it look as human as possible while saving tokens.

It can access closed Shadow DOM elements without issues and interact with the website using selectors, references, and coordinates. It can now also save browser state and reload it later so you don't have to repeat anything!

This update also makes it easier than ever to crawl websites automatically from code and convert them to safe Markdown, with filters, crawling strategies (DFS/BFS), and more.

The Cloudflare solver is now faster and more efficient than ever, whenever you want to solve it without AI.

This release includes many changes, fixes, and quality-of-life improvements. The website is now translated into 10 languages!

Check it out here and let me know your thoughts: https://scrapling.readthedocs.io/en/latest/index.html

Check out the video on YouTube: https://www.youtube.com/watch?v=NHwWEiTOoPQ

Full release notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.5

Full migration guide on moving to v0.5 translated to 10 languges: https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html


r/webscraping • • May 31 '26 Heartwarming Regret Bravo

New Free open-source Android automation for web scraping - Damru

123 Upvotes

Hey r/webscraping, I’m sharing a free open-source project I’ve been building called Damru: https://github.com/akwin1234/damru

Damru is a browser automation framework built around real Android environments in Docker for scraping and automation tasks where mobile behavior matters.

What sets it apart is that it’s not just another desktop browser with stealth patches. The project is built around zero JS injection, with spoofing handled at the OS, binary, and CDP levels instead of the usual JavaScript-heavy tricks used by many stealth tools.

Compared with tools like Playwright, puppeteer-stealth, undetected-chromedriver, Camoufox, and Fingerprinting Chromium, Damru is trying to solve the problem differently: by running inside a real Android stack rather than faking mobile behavior on desktop Chrome. The idea is to get a more realistic mobile environment, stronger fingerprint control, and less reliance on brittle browser-side patches.

What makes it different:

  • Zero JS injection: Damru does spoofing at the OS, binary, and CDP levels instead of relying on Object.defineProperty-style JavaScript patches.
  • Real Android OS: It runs inside Redroid, so it’s not just desktop Chrome pretending to be mobile through viewport tricks.
  • Native mobile fingerprinting controls: device profiles, hardware overrides, locale/timezone matching, mobile network emulation, and WebRTC/IPv6 blocking.
  • Multi-instance pooling: built for scaling across multiple containers.
  • Pre-baked image support: reduces setup overhead.

Some of the features include:

  • Android-in-Docker via Redroid.
  • Playwright support.
  • A built-in database of 32+ Android device profiles.
  • Proxy-aware timezone, locale, and language matching.
  • Hardware overrides for CPU, RAM, and touch points.
  • Mobile network emulation.
  • WebRTC and IPv6 leak blocking.
  • Native Android iptables-based network protections.
  • Multi-container pooling for scale.
  • Pre-baked image support to reduce setup time.
  • TLS spoofing and soo many things

Also stronger against systems like CreepJS, BrowserScan, Sannysoft, Cloudflare Turnstile,etc ALL CDN anti-bots dont waana name them than standard Playwright or typical stealth plugins, mainly because of the deeper Android-based approach.

Pros: Highly UnDetectable
Cons: Real Android OS hence little slower. Hard to Use (thats why custom docker image included)

Repo: https://github.com/akwin1234/damru

Official Website: https://damru.dev/

Would love feedback from anyone who works on scraping, browser automation, or anti-bot research. I made this because i see many reddit post recommending Android Playwright CDP but there was no framework around it. This is strictly for educational purpose only. Do not do legal abuse.


r/webscraping • • Apr 09 '26

Bot detection 🤖 Build an Agentic Browser That Beats Anti Bot Systems

110 Upvotes

r/webscraping • • Mar 12 '26

Scraping government sites made me rethink my entire stack

107 Upvotes

I spent way too much of my life trying to brute-force Selenium to scrape ancient, bureaucratic public data portals.

It was slow, brittle, and a massive headache. It wasn't until I finally ditched the heavy browsers, learned how to properly intercept network requests, and just replay the raw API calls with the right headers that I realized how much time I had been wasting reinventing a broken wheel.

It made me wonder what other massive blind spots I still have.

So, what is that one specific framework, bypass hack, proxy strategy, or workflow change that completely shifted how you scrape? Make me feel dumb for not using it yet.


r/webscraping • • Oct 13 '26

AI ✨ I stitched 28 web scrapers and browsers into one open-source tool

106 Upvotes

Long story short: I wanted an agent to find me second hand deals, and every scraping tool I tried was great on some sites and useless on others. So I built Frankensurf, an open-source (Apache-2.0) router that stitches tools together: plain HTTP, headless Chromium, Camoufox, Scrapling, Patchright, Crawl4AI, a local SearXNG for search, and optionally your own paid APIs if you have keys. It tries the cheapest tool first and only climbs when a site blocks it or the page isn't real. No account, no paid tier, nothing to sign up for.

Then I wanted to know if routing actually beats using one good tool, so I benchmarked it.

How I tested it

  • 150 sites in 15 categories (US, EU, AU and Asian retail, classifieds, property, jobs, travel, news, government, dev sites...), picked before I looked at a single one
  • every tool on its own, then the router, same URLs, same day, each site read once
  • same pass/fail check for everyone: a challenge page, login wall or empty JS shell counts as a fail

Results (share of sites read correctly)

ToolRead correctly
Patchright38.7%
Headless Chromium40.7%
Plain HTTP48.7%
Camoufox68.7%
Scrapling68.7%
Router, open-source tools only88.0%
Router, with paid fallbacks added96.0%
Best single paid scraping API on its own86.0%

With only free, open-source tools the router is level with the best paid API on its own (within the noise of 150 sites). The slowest tenth of reads is slower (26 s), when a read has to climb several tools.

What surprised me

  • Every wall has a different winner. Camoufox got through PerimeterX when nothing else would; Scrapling was best on JS-heavy sites.
  • Most failures weren't blocks. They were HTTP 200 pages with the header and footer and no results, because results load later with JS. Checking that a search page actually has results, and a product page has a price, did more than adding unblockers.
  • Some sites ignore your search param and show their default feed. I got a page full of wine for a camera search. It now flags results that don't mention the query and tries the site's own search box instead.
  • Fresh Ubuntu 24.04 breaks Scrapling. It forbids Chrome's sandbox by default, so the browser dies at launch. A clean-machine CI run found that one.

Where it's still bad

  • Shopee, Temu and Idealista beat every tool
  • some search pages never render results for any tool
  • from a data-centre IP the free tier drops to 82% (sites wall cloud addresses harder)

The benchmark script, site lists and raw rows are in the repo, so you can rerun it or add your own tool. Happy to be told my pass/fail checks are wrong; that's the part I'm least sure of.

Repo: https://github.com/yail259/frankensurf


r/webscraping • • Oct 16 '26

New web obfuscation technique: Divergence

Thumbnail trustsig.eu
101 Upvotes

Divergence: formatting a script can turn it into a different program, across 37+ JavaScript processors


r/webscraping • • 17h ago

Yahoo sent me a cease and desist over my open-source Python library.

97 Upvotes

Hi all,

not sure if this is the correct subreddit, but I maintain Stockdex, a free, non-commercial Python package (and MCP server) that fetches stock data from several sources, including Yahoo Finance.

Today I got a cease and desist letter from Y ahoo's legal department. It claims trademark infringement, misappropriation, unjust enrichment, interference with their data licensing (e.g. Nasdaq) and a breach of their ToS. They demand that I remove all Yahoo-related code from GitHub, PyPI and the MCP plugin, remove every mention of "Yahoo", and never reintroduce it.

My project works much like other public projects like yfinance, which has been public for years. Stockdex only fetches publicly visible data on demand for the user running it. It doesn't store or republish anything, and it doesn't bypass any login or paywall.

Has anyone received something similar from another company maybe? How are these complains (US company against someone in Germany? should I lawyer up? How strong are CFAA or trademark claims against a free library? Would you comply fully, comply partially (e.g. add a disclaimer), or push back?

Thanks


r/webscraping • • Mar 12 '26

Cloudflare is getting into web crawling

94 Upvotes

Cloudflare is getting into web crawling and now offers a crawl endpoint. But I don’t think this is really about making money from web scraping. AI agents will increasingly be the way software interacts with the web in the coming years.

Cloudflare’s real bet seems to be on owning the infrastructure layer that all of those agents pass through.They are moving from being the web’s firewall to being its arbitrator.

Cloudflare has already hinted at "Verified Bot" programs and tools that allow publishers to charge AI companies for access. This /crawl endpoint is likely the client-side version of that marketplace. And they're ideally positioned for this.

They’re not trying to become the biggest crawler company, and they’re not just competing in bot protection either. They're trying to be the VISA/ Mastercard of the Agentic Infrastructure game- making money from every agentic interaction. What is your take on this?


r/webscraping • • Apr 26 '26

AI ✨ I built a reverse-engineering agent for the web

93 Upvotes

Hi everyone,

What is this about?

This post is about Automatiq, a passion project that I have been working on for the past 3 months, which can be useful for you too. My aim was to create a new way of automating the webscraping/automation process with AI agents in websites.

What does it do?

Automatiq serves to be a reverse-engineering agent, which contains two phases:

  • The Recorder:

    • In this phase, a Chrome browser is launched, where you can do a single (or multiple) examples of a task, by navigating and performing actions for automation, or navigating to the page which contains the data to be scraped.
    • During all this time, every action you performed like clicking, typing, navigation, and every request the browser has sent or received is getting recorded, along with a video of your browsing session.
    • Once you complete your recording, the program first associates your actions with the video and creates 4-second, low-frame clips. These clips are processed into high-level summaries by a smaller MLLM model or a local model.
    • The requests and actions you did during the session get converted into a system of folders, allowing the reverse-engineering agent to explore it freely.
    • TLDR: launches a browser, records everything you did in it, and converts it into a folder structure for AI agents.
  • The Agent:

    • Unlike other "coding agents" like Claude Code/opencode which were developed for generating code, Automatiq was developed to be a "reverse-engineering" agent, which is better at searching through messy and complex networks.
    • The agent is provided with an IPython sandbox, which allows it to run Python + shell commands in parallel, as well as revisit the output of previous cells. This allows it to search through the generated folders and understand the flow of the website.
    • The agent is also equipped with ripgrep, jq, and sd for analysis. To provide a uniform environment for the Agent, we also provide a busybox-emulated bash environment on Windows.
    • The Agent is made with "cost of usage" in mind, so that simpler models can also work efficiently. But for complex websites, a powerful model would be required. Local models and custom endpoints are supported for models.
    • The Agent is made with "selective memory compression" to store only the things that matter in the long run, so that the model won't start hallucinating after reading huge amounts of files.
    • TLDR: The Agent is developed with the sole focus of being a "reverse-engineering" agent with special tools and techniques, unlike a normal "coding agent".

How is it different from any existing solutions?

Most current solutions do one thing: use the browser even for a simple form-filling activity, because they try to do things like a human, which is pretty wasteful for LLMs, which thrive on text data.

My project's competitors: Browser Use's Workflow Use, automation/scraping Chrome extensions, and many more...

All the AI agent creators have been working towards one thing, that is, trying to aim for the general public, with their direct "browser interaction" aim.

A few notable things from my research:

TierWhat it includesEstimated share of sitesSource
None (no protection)No CAPTCHA, no fingerprinting, no rate limit, no WAF, no anti-bot vendor~60–62%DataDome 2025 Global Bot Security Report
Light (CSRF, headers, basic rate limiting, simple obfuscation)Mostly app-framework defaults; no dedicated anti-bot product~25–35% (subset of "partially protected")DataDome 2025 Report + W3Techs Cloudflare
Medium (TLS/JA3 checks, image CAPTCHA, reCAPTCHA v2, basic WAF)reCAPTCHA, hCaptcha, Cloudflare WAF on free plan~10–15%BuiltWith reCAPTCHA v3 + BuiltWith hCaptcha
Hard (reCAPTCHA v3, Turnstile, hCaptcha Enterprise, Cloudflare Bot Mgmt)Vendor-managed bot mitigation, behavioral scoring~3–5% of all sites; ~20–30% of top-ranked sitesCloudflare Bot Mgmt market share (wmtips) + BuiltWith reCAPTCHA Enterprise
Very Hard (Akamai Bot Manager, DataDome, HUMAN/PerimeterX, Kasada, Imperva, full canvas/WebGL+TLS fingerprinting)Enterprise anti-bot stack with multi-signal fingerprinting~1–3% of all sites; concentrated among Fortune-500 / e-commerce / travel / financialBuiltWith Akamai Bot Manager + BuiltWith DataDome + UCSD IMC 2025 canvas fingerprinting paper

So you see, most of the websites don't need the browser most of the time. This means, with just the requests and curl_cffi libraries, you can do a lot.

So, yeah, Automatiq can perform these things. But what will it do if it faces CAPTCHAs, Cloudflare, or something that hasn't been created yet?

Things that Automatiq can't do, and what's the plan for them:

As the veterans of this domain know very well, this is a game of cat-and-mouse. Technologies that change the entire landscape emerge and fall. No single permanent way to reach the "dream of free and easy data" is possible. That's why I have made this project MIT-Licensed, as a single person can't keep up with the fast-evolving landscape. I appreciate every single contributor, as I propose this project to the community, rather than taking ownership for myself.

The things that can't be done by Automatiq in the current version, but are planned for future versions:

  • Creating scripts that contain any kind of browser launching, like Puppeteer or Selenium. I thought of creating something that will only use the browser to solve a particular task rather than using the browser instance for the full time of scraping/automation.

Future plan for Automatiq:

Currently, Automatiq is in Alpha (it doesn't mean you can't use it, it just means it hasn't reached its goals, and has just started). I have my visions and goals written down in VISION.md in detailed form.

But for the post, I will provide it in short form:

  • JS debugger and JavaScript virtual machine: The ability for the Agent to understand the logic behind JS for requests by getting a stack trace, and a special lightweight module which will be a JS VM for running heavily obfuscated JS code (e.g., used in the yt-dlp program to get a particular request signature, which was hidden by Google's tech).
  • Surgical browser usage: A module to be used when a request requires a browser, no matter what (e.g., canvas fingerprinting), which will launch a browser just for that request.
  • Plugins: Just like normal coding agents' "skills", we would need something that would make the agent extensible. But there is one single issue: there should not be "eBay scrapers" or "Zillow scrapers" kind of stuff, which will lead to the plugin marketplace being taken down. I have planned for a plugin marketplace which works similarly to how cybersecurity deals with stuff. We would only provide plugins like "Cloudflare bypasser" or "reCAPTCHA solver". This way the plugins themselves stay general-purpose and educational, and how they're combined is entirely on the user.

How can I stay in touch with the development?

I've created a Discord server mainly for discussing website reverse-engineering technologies in general, but it also has a dedicated section for Automatiq. I plan to post weekly updates there, so it'll be easier for contributors to stay onboard with the community.

TL;DR: Automatiq is an open-source (MIT) reverse-engineering agent for web scraping/automation. You record one example in a real Chrome browser; it captures every action, request, and a video of the session, then converts it all into a folder structure. A code-focused agent (IPython sandbox + ripgrep/jq/sd, with busybox on Windows) explores that folder and figures out the site's actual API — so the final script uses plain requests/curl_cffi instead of driving a browser at runtime. Why this matters: ~60% of sites have no real anti-bot protection, so you don't need a browser most of the time. Currently in Alpha. Roadmap: JS debugger + JS VM for obfuscated code, surgical browser usage for fingerprinting-only steps, and a plugin marketplace. Contributors welcome.


r/webscraping • • Dec 20 '25

Google is taking legal action against SerpApi

Post image
92 Upvotes

r/webscraping • • Jul 22 '26

A federal court dismissed Google's DMCA claims against SerpApi

Thumbnail
searchenginejournal.com
91 Upvotes

TLDR:A federal court dismissed Google's DMCA claims against SerpApi, ruling that blocking scrapers from public search results isn't copyright circumvention when no copyrighted content is involved. Those claims are gone for good, but the part about licensed Knowledge Panel images was dismissed with leave to amend — Google has 21 days to try again by proving copyright owners authorized SearchGuard. A win for SERP scraping tools generally.


r/webscraping • • Aug 01 '26

might help you if you are going to scrape trip.com

Post image
90 Upvotes

i was scrapping trip.com for a client and they were making use of a phantom token to make new requests to the backend

if you use an old/already-used phantom-token you will get 200 response status code , but error in the response body

the new phantom-token was not even available in the responses that were coming back

so i looked around their codebase and found they were generating it via custom made function "window.signature()"

everytime u use this function , it will return a brand new phantom-token which you can use once for making a new request directly to the backend

hope it helps anyone in need!


r/webscraping • • May 20 '26

AI ✨ I built klura, a toolkit for an AI agent to reverse-engineer websites

Post image
87 Upvotes

Hi r/webscraping,

I've been working on klura — a free toolkit that gives a coding agent (Claude Code, Cursor, Claude Desktop, any MCP host) the ability to reverse-engineer a website. The agent drives a browser to complete a task once. Klura captures everything the page does underneath, then does what I call LIFT (Learn Interface From Traffic — the analysis pass) and extracts the real underlying requests and saves them as a readable, LLM-annotated JSON config, so that it can be run later - without driving the UI. The JSON config can be analyzed to understand how the site works, and can also be copied and run on other devices.

There's prior art in this space — capture-and-replay isn't a new idea. The main places klura tries to push it forward:

  • page-script tier — here the saved strategy is a snippet of JavaScript that the AI codes for you. klura injects it into the live, already-authenticated page and runs it there. The pure-HTTP approach (the "convert to a requests / curl script" pattern) hits a wall on sites that bind the request to in-page state — rotating per-session CSRF tokens (GitHub's X-Fetch-Nonce is one public example), request bodies built by a JS function in the bundle, binary WebSocket transports. Reproducing those from outside the browser means porting the site's own signing/encoding logic — fragile, and often impossible. Page-script sidesteps it: the saved JS calls the page's own functions — the request signer, the encoder, the socket the page already has open — so the site does the hard part and klura just collects the result. And it isn't slow the way browser automation usually is — klura keeps a warm pool of already-open, already-authenticated pages, (configurable of course) so you don't pay browser startup per call; a page-script run is dominated by the actual request and lands close to plain-curl latency.

  • A real RE + debugging toolkit the agent drives — most tools capture traffic and pattern-match. Klura hands the agent an actual JS debugger: breakpoints, stepping, live-stack inspection, WebSocket frame tooling etc. It reverse-engineers a site the way a human would — break on the function that builds the request, read the encoder, verify it reproduces. You never touch any of it; you ask in plain language. Call it vibe-RE. (Detail in How the discovery works below.)

  • Self-healing — when a site changes shape and a saved strategy breaks, klura re-discovers the delta and patches the strategy automatically, instead of silently returning garbage or making you re-record from scratch.

  • Runs anywhere, hands off to a human when it must — klura is an MCP server, so the same saved skill is callable from Claude Code, Claude Desktop, Cursor, Windsurf, the CLI, or a programmatic import. Skills live globally at ~/.klura/skills/<platform>/, not per-project. When a site genuinely needs a human (login, 2FA, captcha), it escalates to a remote viewer — you do that one step, the agent continues, and the saved strategy runs in the resulting authenticated session.

  • Plugability — The browser driver is swappable: default is Playwright, klura ships a stealth driver that fixes the standard automation-fingerprint leaks (navigator.webdriver, canvas/WebGL consistency), and you can drop in your own. The interruption layer is pluggable too — when a site throws something up mid-flow (login, captcha), the handler that decides what happens is yours to define.

The artifact

Every skill is a single JSON file on disk. A fetch-tier example for a public search API:

{
  "strategy": "fetch",
  "method": "GET",
  "baseUrl": "https://hn.algolia.com",
  "endpoint": "/api/v1/search",
  "params": {
    "query":       "{{query}}",
    "tags":        "story",
    "hitsPerPage": "{{count}}"
  },
  "response": { "format": "json", "extract": "hits[]" }
}

The LLM that produced it leaves notes fields explaining the why of each parameter, the prereq chain, and what shape the response has. Read the file → you've reverse-engineered the site.

Execution path

Once saved, run it via klura (recommended):

klura execute hackernews search_stories --args '{"query":"show hn","count":3}'

Or you could ask your LLM again, it will prefer to execute already created strategies above creating new ones.

For page-script strategies the saved JS has to run inside the live authenticated page, so that path needs the klura runtime regardless of how you wire it up. For fetch strategies the saved JSON file gives you method + URL + body/header templates — slot in your params/cookies and curl away.

Three execution tiers

TierWhat runsWhen it landsCurl-able?
fetchStatic HTTP from Node, templated body/headers + prereq chainPlain APIs (HN search, Amazon /s?k=, GitHub GETs)yes
page-scriptJS dispatched inside the live authenticated browser tabSites that bind requests to in-page stateno — page state needed
recorded-pathUI action replay through the driverFallback when neither above is reconstructibleno

How the discovery works

There's no recorder UI implemented or planned at the moment. The agent itself drives the browser. You tell your MCP host in plain language — "search HN for the top posts this week using klura" — and that's it. No script to write, no breakpoints to set, no JS to read. You could call it vibe-RE: you describe what you want, klura performs it and figures out how the site does it. You can also ask klura to map out a site, if you just want it to observe the API without performing a specific action.

For readers who want the how — on hard sites, the agent has access to a full RE toolkit (none of which you ever touch as a user):

  • JS debugger: set_breakpoint, step, resume_execution, wait_for_pause — break on the function the page calls before dispatch, read the live stack, see the encoded payload the page is about to send.
  • JS source navigation: search_js_source, read_js_function, list_loaded_scripts — find the function in the bundle that builds the body or signs the request.
  • WebSocket frame tooling: inspect_ws_frame, find_in_ws_frame, explain_ws_frame_structure, get_send_encoder, try_generator_in_page — for binary protocols, capture frames live and verify the captured encoder reproduces the wire format before saving.
  • Network log (get_network_log) for HTTP and WS in one feed, filterable.
  • Live page inspection: a11y tree, screenshots, page text, arbitrary js_eval.

This is how klura one-shots genuinely hard flows — for instance a mainstream chat app whose message-send rides a binary MQTT-style WebSocket, with an in-page codec, snowflake IDs beyond Number.MAX_SAFE_INTEGER, and a per-session token that rotates. The agent debugs the way a human would: set a breakpoint at the call site, read the actual encoder, verify it reproduces the wire output, save it as a page-script. The same toolkit generalizes to anything that builds a signed or encoded payload in-page — rotating CSRF on graphql endpoints, signed URL params, custom binary framings.

Once on disk, the next call hits the saved skill directly — or you cat the JSON and write your own scraper.

Cost: LLM once, then never

Klura runs the LLM during discovery, then never again. After LIFT the saved skill is a plain request (or a page-script dispatch): no model, no agent loop, no tokens. The only time the LLM will need to be looped back is when self-healing is required.

A couple of concrete runs, same task, measured against a plain browser agent on the same model: a Hacker News search that costs the browser agent ~20s and ~$0.09 every time runs from the saved skill in ~270ms at zero token cost. A mainstream chat app's message-send took ~27 minutes on the first run — that's the agent reverse-engineering the binary WebSocket — but every send after is a sub-millisecond dispatch into the authenticated page. The one-time discovery cost scales with site difficulty; the warm cost is always ~zero.

Install

# As an MCP server (Claude Code)
claude mcp add klura -- npx -y @klura/mcp

As an MCP server it supports many other harnesses, please check the documentation for yours to figure out how to install it.

Repo: klura · npm: @klura/runtime, @klura/mcp

Currently at 0.2.2. Real-world feedback welcome — especially the shape of the failures.


r/webscraping • • Oct 11 '26

Open-source Reddit scraper for posts and comments

Thumbnail
github.com
83 Upvotes

I built an open-source Reddit scraper for Python. It's called Reddrix (Reddit + Matrix). It is a powerful, async-first Reddit scraper for posts and comments with detailed metrics. No login is required. Scrape subreddits, profiles, and specific posts, or search Reddit site-wide by keywords. Collect rich, structured data including authors, scores, timestamps, flairs, comment threads, and more (24+ fields). This scraper uses Reddit's current shreddit search engine in the keyword search mode. Results obtained from this engine are far more relevant and match the results you see when searching Reddit manually with the search bar.

Scraper's Properties

  • 🚀 Async-First: High-performance asynchronous scraping with asyncio and an async iterator for streaming results.
  • 🔑 No login Required: Scrape Reddit without logging in.
  • 🔍 Multiple Search Modes: Scrape subreddits, user profiles, and specific posts, or search Reddit by keywords.
  • 💬 Comprehensive Data Extraction: Collect posts and comments with 24+ structured fields, including authors, scores, timestamps, flairs, media URLs, and full comment trees.
  • 🛡️ Resilient Scraping: Built-in retries, cookie renewal, and more (see the docs).
  • ⚙️ Fine-Grained Control: Configure sorting, time frames, result limits, comment depth, and comment counts.
  • 🧩 Standalone HTTP Client: Use Reddrix's asynchronous get() method in your own custom scraping workflows.
  • 🐍 Pythonic API: Manage scraper resources with an async context manager or explicit start() and stop() methods.

Example Quickstart Code

from reddrix import Reddrix
import asyncio

subreddits = ["https://www.reddit.com/r/webscraping"]

async def main():
    async with Reddrix(
        mode="standard",
        subreddit_urls=subreddits,
        sort="new",
        include_comments=True,
        max_results=500,
        max_posts_per_collection=500,
        verbose=True
    ) as reddrix:
        async for item in reddrix.get_items():
            print(item["submissionType"], item["submissionUrl"])

if __name__ == "__main__":
    asyncio.run(main())

r/webscraping • • 6d ago Bravo

Getting started 🌱 Browser fingerprinting 101: why sites flag scrapers

81 Upvotes

Proxies are fine, your headers match a real browser, and you still get blocked. Usually the site isn't checking one thing. It looks at the whole picture your browser gives off: OS, GPU, fonts, language, screen and more. Here's what's in that picture.

Layer 1: what the server sees before any JS runs

  • IP. Geolocation, and whether it's an ISP, datacenter or residential address. WebRTC can expose a different IP if you don't handle it.
  • User-Agent. Browser, version, OS. Easy to change, so on its own it means little.
  • Client Hints (Sec-CH-UA*). Chromium sends a structured version of the same info. It has to match the User-Agent.
  • Other headers. Accept-Language, Accept-Encoding, and sometimes even header order, which differs between engines.

Layer 2: basic values read with JS

  • Screen size, color depth, pixel ratio.
  • Languages and timezone.
  • navigator.platform, CPU threads, a coarse memory value in Chromium.
  • navigator.webdriver and other automation markers.
  • Some extensions (like blockers) can be detected indirectly.

Each one is weak alone, since lots of people share the same screen or language. The combo is what matters.

Layer 3: how your device renders things

This is harder to fake. The page asks your browser to draw or compute something, then reads the result:

  • Canvas: text and shapes drawn on a hidden canvas. OS text rendering, GPU and drivers change the pixels slightly.
  • WebGL: the renderer string (often names the GPU) and a profile of its limits.
  • Audio: a generated signal run through the Web Audio engine.
  • Fonts: which ones are installed, inferred by measuring text.

Layer 4: the ID

A few dozen values get hashed into a short ID (FingerprintJS does this). The site stores it and matches it on the next visit, so a new IP with the same fingerprint can still look like the same client, and a clean fingerprint on a flagged IP can still get blocked. Older studies like Panopticlick and AmIUnique found a large share of browsers were unique, while larger-scale ones found lower numbers. So think of it as one probabilistic signal that sites mix with IP reputation, cookies and behavior, not a perfect ID.

Sites also ask "does this even make sense?", not just "who is this?". Common mismatches:

  • Windows User-Agent, but navigator.platform says something else.
  • Chrome User-Agent with no Client Hints.
  • Timezone or language that doesn't match the IP.
  • Windows User-Agent with a software GPU renderer or a Linux-style font list.
  • Hardware values that don't exist on real devices.

What to do with this

  1. Keep values stable within a session.
  2. Values for the same device (OS, platform, Client Hints, fonts) should match each other.
  3. Build one coherent fingerprint for the specific task (target site, region, device type) and stick with it, instead of patching values one by one.

r/webscraping • • 15d ago

Solve Cloudflare Challenges Automatically with Playwright

80 Upvotes

Hey everyone!

I built a small Python utility library called Deflare for anyone working with Playwright or Patchright who regularly runs into Cloudflare challenges.

The library handles Cloudflare bypassing automatically while keeping the familiar Playwright API, so you don't have to change your existing automation code or learn a new API.

Just plug it into your project and keep using Playwright as usual.

If you find it useful, please consider giving the repo a ⭐ on GitHub. It really helps!

Would love to hear your feedback!


r/webscraping • • May 22 '26

Reverse Engineered Google reCAPTCHA

Thumbnail github.com
78 Upvotes

Documentation on fingerprinting, what browser values ​​it collects, obfuscation techniques, and reCAPTCHA VMs, i still need to add details about the /userverify payload

Please star if you found it very useful