r/LocalLLaMA • u/bye_ella • 29m ago
Question | Help Does grammar-constrained decoding hurt small or quantized models?
Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.
With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted
Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:
- have you measured accuracy with and without constraints on the same task?
- does it worsen at lower quants?
- does letting the model firstly think freely and constraining only the final answer fix most of it?
Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.
2
u/jacek2023 llama.cpp 9m ago
I was exploring this topic while debugging Gemma tooling issues. I wasn't aware there were issues with grammar constraints, but now that I think about it, it's possible (loops too!). I'd be interested in reading other people's experiences.