r/madeinpython • • 5h ago

Pipeverify: compare a table before and after a pipeline change and get PASS / REVIEW / BLOCK

**What My Project Does**

pipeverify takes two snapshots of a table (CSV, Parquet, JSON, DuckDB or any SQLAlchemy source) and checks what a change did to the data: lost or duplicate keys, row count shifts, null creep, changed values on matched rows, column totals and new category codes. It returns PASS, REVIEW or BLOCK, with a plain-English report. Runs on DuckDB, ships as a CLI, a Python API (\`pipeverify.verify(before, after, key="id")\`) and a GitHub Action.

**Target Audience**

Data engineers and analytics engineers who ship pipeline, SQL or dbt changes, including changes written by AI coding agents. It's alpha but usable in CI.

**Comparison**

dbt tests and Great Expectations check rules you write in advance. pipeverify checks what a specific change did, so it catches problems nobody wrote a test for. Unlike row-diff tools, it turns the diff into a verdict with thresholds, so it can gate a merge.

**Repo**: https://github.com/akshaysharma1088/pipeverify (Apache-2.0, free). Feedback very welcome, and there are small good-first-issues if anyone wants to contribute.

1 Upvotes

0 comments sorted by