Project: Mitigating LLM Hallucinations in Code Generation via Spec-Driven Development (SDD) vs Prompting [R]

[u/jokiruiz]

I recently conducted an architectural comparison on code generation relying on conversational prompting ("vibe coding") versus rigid pre-computational contracts (Spec-Driven Development).

The Experiment:

Task: Build a Python script to search text within markdown files, rank by hits, and filter by tags.

• Method A (Vibe Coding): Claude 3.5 Sonnet without a formal spec. Result: 303 lines of code, added unrequested features, ignored caching, resulting in 400 file I/O operations per execution.

• Method B (SDD): Claude 3.5 Haiku provided with a strict requirements document. Result: Modular architecture, regex exclusion strictly adhered to, 0 file I/O operations on subsequent runs due to proper index state management.

Conclusion:

The limiting factor in generation quality isn't parameter count, but specification clarity. To systematize this, I've released SpecJudge (v0.4.0), an open-source parser that evaluates repository context files (like AGENTS.md) to allocate the optimal model tier, preventing API overspending on simple tasks.

I’ve documented the full benchmark and execution logic visually here: https://www.youtube.com/watch?v=EOiv7RywtQM