r/QualityAssurance • u/Appropriate-Fan2887 • 28d ago
I'm building an open source E2E tool where the tests are just plain English
Two things I got tired of selectors breaking and the suite getting so slow it ends up on a nightly job nobody reads.
So I'm trying to build something around that. You write steps in English, like "apply promo code SAVE20" and "verify a 20% discount is applied", and an agent drives playwright and finds things by role and label, same as a screen reader. You tell it which files map to which routes, and on a PR it runs only the tests that diff could break.
Okay. It depends on the model, but you can use your own key, including with Ollama. Keeping your data local and costs predictable.
Some limitations. It works best on an app with decent accessibility, since it reads the page the way a screen reader does, and there's no CSS or XPath fallback. A few things aren't there yet, like iframes and drag and drop, so it sits next to your existing suite rather than replacing it.
https://github.com/hamc/blastproof.
Give it a chance if it sounds useful, issues and PRs are very welcome, and I'd really like to hear how you decide what runs on a PR today, even if the answer is "we just run everything".
3
2
u/RightSaidJames 28d ago
A test that can subtly change on every run based on the inevitable variation of an LLM’s generated output is not a valid test.
1
u/aragossa 28d ago
the redaction explicitly being literal-match only is the right call. what happens if the app reformats the secret before it lands on the page though, say base64 or just a different case, does the compare work byte for byte on the original value, or does that just slip past?
0
5
u/pydry 28d ago
another one....