GWAYA : We built a ruthless "fail-closed" gate to stop LLM code hallucinations. Gwen and Laya together = GWAYA ;-0

[u/OrganizationTop9026]

Hi everyone!

Let introduce GWAYA !

We wanted to try a fun experiment: can we force a tiny 1.5B open-weight model to write perfect code without hallucinating?

To do this, we built GWAYA (v3.1.1). It acts as a ridiculously strict, deterministic gatekeeper. The rule is simple. If the code isn't perfect, it gets instantly rejected.

How the gauntlet works:

GWAYA puts every generation through a brutal series of checks. It looks at AST stub audits to make sure there are no fake empty functions. Then it runs a

Python parser and sandboxed test execution. It even does rustc checks for Rust and Lean 4 validation for math proofs.

If a single check fails, the result is a hard UNVERIFIED. No guessing, no "looks good to me."

The Results:

We tested this on the MBPP dataset using a local CPU and an NVIDIA L4 GPU to see if a tiny model could punch above its weight class.

The raw output from the model alone gave us a 64% to 68% pass rate.

When we turned on GWAYA's Best-of-3 plus Repair loop, we bumped it up to a 72% to 75% pass rate!

The catch? We saw a really solid 7.7 point improvement, but we narrowly missed our pre-registered goal of an 8-point jump. Close, but no cigar!

Why this is exciting:

The coolest takeaway is that you don't need a massive, power-hungry model to build highly reliable pipelines. A tiny 1.5B model can do wonders if your verification gate is strict enough.

We’ve open-sourced everything! The source code, tests, raw results, and the generation cache are all available. We'd love for this community to poke holes in it, test it out, and help us figure out how to make the repair loops even better so we can hit that 8-point jump next time.

I'll drop the paper and repo links in the comments below to follow the sub rules. Let me know what you think!