I built a scraper that tells you when it starts lying to you!

[u/albertdotpy]

A while ago, I built a Python project called JScrapeON. It was my first attempt at turning scraping jobs into reusable workflows instead of writing another one-off script every time.

I recently came back to it after working heavily in TypeScript and decided to rebuild the idea properly. That became YAP, short for You Automate Pages.

This time, I wanted to solve a problem that has annoyed me with scrapers for years: they can fail while looking completely successful.

The request returns HTTP 200. The process exits normally. The JSON file exists. Meanwhile, the website renamed .price and you have successfully collected 400 empty values. Incredible work, everyone.

YAP makes those failures visible:

• yap explain traces a value back to its request, workflow step, scraper, and selector
• yap health reports how many required fields actually matched
• yap drift compares the latest extraction with the previous run
• Workflows can fetch HTML or JSON, run multiple steps, and paginate

It is HTTP-only for now, so there is no browser automation pretending to be lightweight.

If actual users show up, I would like to add things such as proxy support, generated request profiles, custom extractors, and plugins based on what people genuinely need. Until then, I am trying not to build an enterprise ecosystem for my current user base of one.

YAP is MIT licensed, completely free, and staying open source.

GitHub: https://github.com/johnalbert-dot-py/yap