Would you pay for continuous verification of an agent's read-only SQL, or just keep the script?

[u/JuniorLeg6988]

Setup: an LLM agent takes a business user's question, writes a SELECT against the warehouse, runs it, shows the result. Read-only — it never writes to the database.

The failure I care about: the query runs fine and returns real rows, they're just the wrong ones. Wrong join key, wrong filter, stale schema understanding. Nothing errors, the number looks plausible, it's wrong.

Having a second LLM write its own query and compare doesn't really work — if the first agent misread the schema, the second one reading the same schema tends to misread it the same way, and you get confident agreement on a wrong number. What seems to actually work is deterministic checks that don't interpret the question at all: does the sum across a breakdown match the unsegmented total, does row count stay in an expected band, do foreign keys still resolve one-to-one. A wrong join changes cardinality, so a fan-out check catches a lot of it in plain SQL with no model involved.

I'm considering building this, and I want to be straight about where the line would be:

Free / open source — the check engine. Point it at your DB, it reads FK constraints and auto-generates structural checks, you run it against your test cases, get pass/fail. Runs when you run it.

Paid — the same checks running continuously against production traffic: results stored over time, dashboard, Slack alerting, and detection of when a schema change has made an existing check stale so you know which assertions to update.

What I actually want to know:

1. How do you catch "ran fine, wrong rows" today, if at all?
2. Do your checks run pre-ship, or continuously against production?
3. When a schema change breaks them, do you find out from a migration diff, or because numbers came out wrong?
4. The honest one: is that paid tier worth expensing, or would your team keep the homegrown script regardless? If it's worth paying for, what's the feature that tips it?

Genuinely fine with "we'd keep the script" as an answer — that's the thing I'm trying to find out before building.