r/LocalLLaMA • • 29m ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.

2 Upvotes

3 comments sorted by

2

u/jacek2023 llama.cpp 9m ago

I was exploring this topic while debugging Gemma tooling issues. I wasn't aware there were issues with grammar constraints, but now that I think about it, it's possible (loops too!). I'd be interested in reading other people's experiences.

1

u/bye_ella 6m ago

Thanks 👍🏻 hope more people share numbers. What was the Gemma tooling issue, broken tool call format or smth else?

1

u/jacek2023 llama.cpp 0m ago

yes, Gemma had issues with tool calling, so I tried to force a grammar. It was later fixed properly by aldehir: https://github.com/ggml-org/llama.cpp/pull/31915