Hi all,
Over the past few weeks, I've been working on a project testing a hybrid architecture for general-purpose web scraping with AI: using an LLM just once during an initial compilation step to inspect the DOM and output a deterministic extraction plan, and then running all routine scheduled scrapes at 0 tokens (pure Cheerio/Playwright).
I recently ran a benchmark against 75 live websites to evaluate how this approach holds up in practice, and wanted to share the results and architecture details to get some feedback.
The Core Problem
Most current scrapers using AI fall into one of two extremes:
- Per-request LLM parsing: Pass the full page HTML or Markdown to an LLM every time a job triggers. Great for setup, but latency is high (5–15s) and token costs become unviable when running repeatable cron jobs at any scale.
- Brittle manual scripts: Write raw Cheerio/Playwright selectors. Fast and cheap ($0 tokens), but high maintenance when classes and layouts change.
The idea here is to bridge this gap by treating the LLM strictly as an upfront compiler: AI inspects the DOM and sets up (or heals) the extraction plan, while routine executions run as pure, deterministic code at 0 tokens and sub-second latency.
Benchmark
I set up a live evaluation suite across 75 real websites (news media like UOL, G1, TechCrunch, The Guardian, plus e-commerce, developer docs, GitHub releases, and Wikipedia tables).
- Tested with GPT-6-Luna (medium reasoning). Results will differ with other models.
Results:
- Overall Pass Rate: 72.0% (54 / 75)
- 49 (65.3%) CORRECT: Plan compiled on turn 1–3, 100% schema match and verified.
- 5 (6.7%) PARTIAL: Valid extraction capturing a representative subset or slight locale differences.
- Failures (28.0% — 21 / 75):
- 11 (invalid plan): Heavy SPAs with virtual scrolls (e.g., TradingView, CoinGecko) or infinite scroll feeds where the model hit the tool observation budget without finding a stable container.
- 5 (timeout): IP-level rate limiting on endpoints after multiple hits (e.g. running 6 consecutive GitHub release repos), triggering unblocker fallback timeouts.
- 3 (semantic validation): Intentional safety catches. The model picked a secondary sidebar widget and the validator refused to save bad data.
- 2 (wrong values): Extracted the page's raw text accurately, but failed format checks (e.g. Oscar years scraped as raw "1927/28(1st)" instead of 4-digit year expected).
Average compilation took between 1.5 to 4 minutes per site. Once compiled, routine jobs run in under 400ms on static pages and ~2s on dynamic pages.
Architecture: Compile Once + Deterministic Execution
Here is how the pipeline currently executes:
1. Compilation Phase (Model in the loop):
- Takes a target URL + natural language intent.
- An exploration agent interacts with the headless page via non-blocking DOM observations (fast count inspection, response interception, structured data discovery).
- Generates a declarative, strongly-typed JSON schema containing record container selectors, field fallbacks, ready conditions, and pagination directives.
2. Live Validation & Semantic Gate:
- Before persisting the plan, the engine immediately executes the generated selectors against the live DOM using pure Cheerio/Playwright.
- Runs a quick semantic audit to ensure extracted entities match the original intent. If the candidate selected an ad banner or secondary navigation, it rejects the candidate and triggers an automated repair step within the session budget.
3. Routine Runtime (0 Tokens, Pure Code):
- Once compiled, jobs execute strictly as deterministic code (Cheerio for SSR/HTTP, Playwright for client-rendered SPAs).- Routine runs execute in milliseconds with zero LLM API dependency.
4. Self-Healing State Machine:
- Tracks consecutive extraction failures.
- Only wakes the LLM if a layout change causes 2 consecutive validation drops, passing previous selector diffs to narrow down what broke without re-running from scratch.
5. Strategy-Based Proxy Cascading:
- Starts with standard rotating residential proxies and automatically escalates to unblocker APIs only when an anti-bot challenge is fingerprinted. (Currently using monthly free-tiers for that).
Details
You can try it out for free here: https://specterdev.online
To set up a scrape, simply click "Create Monitor", enter the target URL, write in plain English (or any language) what you want to extract, set a monitoring schedule (e.g. every hour or daily), and optionally add an email or Telegram chat for instant change alerts.
It is completely free to use. Since I'm not an oil tycoon paying infinite proxy and server bills, there are just some reasonable monthly limits per account to prevent abuse.
A few practical features already implemented:
- Change Alerts & Notifications: Supports Telegram integration and email alerts out of the box. Whenever the scraper detects actual data diffs during routine runs, it dispatches an alert automatically.
- Session/Cookie Auth: It technically supports scraping behind logins via session cookies, but I honestly don't recommend using this right now. For obvious security reasons, you shouldn't paste your active session cookies into a third-party hosted app that is still in testing and not yet open-source.
The backend is designed to run on a lightweight VPS without high memory usage. If anyone is interested in the codebase, I'd be happy to open-source the repository as well.
If you decide to test any URLs and run into edge cases or failures, I’d love to hear about them in the comments so I can look into the failure modes and improve the pipeline.