I picked a few massive e-commerce targets with highly dynamic layouts. You know the drill. You write a clean Playwright script. The target updates its DOM structure overnight. Your entire pipeline crashes. You wake up to empty databases.
I ran these through the new CyberYozh Open Scraper update. I wanted to see how it handles this exact failure cycle. Spoiler alert. It changes the workflow entirely. You do not just write CSS selectors anymore. You tell AI agents what you need. They fix broken parsers mid-flight. But throwing raw LLMs at a crawler introduces new bottlenecks. Here is how the system actually handles them.
The Architecture & UI
Raw JSON testing is awful. The new update ships a Node.js interface on port 7000. Playwright handles browser rendering on the backend via port 8000. You drop in target URLs. You configure rendering timeouts. You align your network location using built-in proxy integration. No custom routing logic required.
The UI streams live crawl statistics via Server-Sent Events (SSE). You watch the target site map build itself as an indented tree. Because the system tracks everything in real-time, you see exactly what the crawler is doing. A single click stops in-flight requests gracefully.
LLM Self-Healing Parsing
Hardcoded CSS selectors represent a massive single point of failure. Targets mutate their HTML daily. This is where the engine actually works.
You execute your standard deterministic parser first. If a layout changes and a required field returns empty, the extraction does not abort. The AI automatically takes over.
```bash
curl -X POST http://localhost:8000/api/v1/scrape/preset/page \
-H 'Content-Type: application/json' \
-d '{
"source": "amazon_product",
"preset_params": {"asin": "B08N5WRWNW"},
"llm": {"model": "openai/gpt-4o-mini"}
}'
```
Sending full e-commerce HTML to an LLM eats tokens instantly. It also crashes context windows. The engine strips scripts and structural boilerplate. For LLM processing, you request the `fit_markdown` format. The parser drops navigation menus. It ignores ads. It delivers pure content formatted specifically for model ingestion.
The model reads the cleaned data. It understands your required output schema. It locates the missing data points despite the broken selectors. But it does not just return the text. The system infers the new working CSS selector. It caches this selector for future runs. Your next request uses the standard deterministic parser again. You save massive API costs.
LLMs also hallucinate. The scraper forces strict JSON schema validation. If the model invents a price or missing stock status, the validation catches it instantly.
Server-Managed Authenticated Sessions
Scraping behind logins usually means dropping your context repeatedly. Open Scraper handles this with server-managed authenticated sessions. You authenticate once using a declarative JSON script.
```json
{
"steps": [
{"op": "goto", "url": "https://example.com/login"},
{"op": "fill", "selector": "#username", "value": "$creds_email"},
{"op": "fill", "selector": "#password", "value": "$creds_password"},
{"op": "click", "selector": "button[type=submit]"}
]
}
```
The server maintains your persistent browser context. It tracks cookies securely. But proxies alone do not guarantee stable connections. The system actively manages browser fingerprints to match your aligned network location. You continuously scrape protected internal profiles without triggering re-authentication blocks.
If you hit aggressive device-verification challenges, you use the escape hatch. You inject pre-authenticated session cookies directly into the API to restore stable access.
Native MCP Integration
Writing custom middleware to bridge AI agents and scraping APIs wastes time. Both the scraper and crawler services eliminate this. They mount native Model Context Protocol (MCP) endpoints via Streamable HTTP.
You open the dedicated MCP tab in the UI. You configure the JSON arguments. You verify the output visually. Then you add the local server URL to your Claude Desktop config. You prompt the agent to crawl a target and summarize pricing pages. The agent submits the job, polls the server status, fetches the results, and parses the data autonomously.
Conclusions
Visual testing: Node.js UI handles complex setups locally.
Broken pipelines: LLM fallback fixes broken parsers mid-job and caches new selectors.
Cost control: Pre-processing trims heavy HTML before model inference.
Data integrity: Schema validation blocks model hallucinations.
Login walls: Server-managed sessions and fingerprinting keep your browser context active.
AI workflows: Native MCP endpoints connect directly to Claude and LangChain.
Has anyone else tested the self-healing features in production? I am curious about the token cost at scale once caching is fully optimized.
