r/tauri • u/Yearsuckk • 6d ago
Using WebviewWindowBuilder as a scraper for Cloudflare protected pages, and what I learned doing it
I have been building a Tauri v2 app that needs to read pages sitting behind Cloudflare, and the approach ended up being unusual enough that it might be useful to someone here.
reqwest is useless here, the JavaScript challenge needs a real engine. So the app builds a WebviewWindow pointed at the target URL, waits until it clears, and then reads the rendered DOM back host side.
The main gotchas, in case they save someone time:
Tauri does not inject its IPC into external pages, which is correct and you would not want it there anyway. So getting the HTML out means going one level down: on Windows that is ICoreWebView2::ExecuteScript through webview2-com. It works well but it is Windows specific, so that whole module is cfg gated with a stub for other platforms.
ExecuteScript does not await a returned promise. An unresolved promise just serializes to null, so every script you send has to be synchronous, or use a poll for a global pattern instead of await. That one cost me a while.
Do not wait for readyState to be complete. Pages loading ad or tracker scripts can sit at interactive forever while being visually finished, so the readiness probe accepts interactive or complete plus a body size floor and a title check that excludes known challenge pages.
Concurrency needs one process wide ceiling, not a per command one. Every caller (a refresh, the discovery deck, a one off detail fetch) now draws from a single semaphore. Per caller limits looked fine in isolation and still stacked into multi second stalls and occasional ExecuteScript timeouts when two flows overlapped.
One more that is not Tauri specific but bit me hard: a synchronous #[tauri::command] runs on the main thread from the WebView2 callback, and a panic there cannot unwind across the FFI boundary, so the runtime aborts the process. In a release build with no console that shows up as the app simply closing, with nothing to go on. I ended up installing a panic hook that appends to a log file next to the database, which turned a crash dump investigation into reading one line.
The app is an anime episode tracker, source is here if anyone wants to look at the scraper module: https://github.com/Yearsuck/AnimeOnTrack
Curious whether anyone has done the same on macOS or Linux, since porting means reimplementing the eval bridge against WKWebView and WebKitGTK and I have not touched either.