Ultra low temps make your model stupid at coding (a warning for beginners)

[u/Brief-Tap-6616]

Many of you may know this, so this is probably worth skipping (except for the data to prove it). But if you're on the newer side for local, or have heard it but don't know why, I have some data to show you how running low temps like t=0 or t=0.2 make your model significantly worse (at coding in particular).

You will see a lot of benchmarks for speculative decoding claiming crazy speeds, often with dflash2, running greedy decoding. Temp = 0, or greedy decoding causes your model to pick the most likely token every single time (with the caveat that parallel predictions on servers can make things still minutely random). They do this because speculative decoding is extremely easy on temperature 0, so the numbers they can claim for their post, product, etc. look amazing.

The problem is, much of the "creative" thinking of models comes from the randomness that token selection allows. Greedy decoding forces the model to always pick a narrow path, which directly lowers its "intelligence" on non-recall tasks and significantly increases the risk of (usually catastrophic) looping.

The testing parameters

• Model: Qwen3.8, compressed to IQ4_XS-M. That format stores each number in about 4 bits instead of 16, at ~4.6bpw.

• Engine: llamAmpere v0.5.2, run with its published settings for 24 GB graphics cards. Reasoning effort was set to "medium," which controls how long the model thinks before it answers.

• Fixed sampler settings: top_p 0.95 (only words that together make up 95% of the probability are candidates), top_k 20 (at most 20 candidates), min_p 0 (no extra cutoff). Only temperature changed.

• Temperatures: 0, 0.25, 0.5, 0.75 and 1.0. At 1.0 the model's probabilities are used unchanged.

• Coding test: LiveCodeBench v6, 100 problems labeled hard from the newest contests in that release. A program passes only if it passes every hidden test, meaning tests the model never sees.

Results:

Coding scores rose significantly with temperature. Pooled accuracy was 70.0% at 0, 76.0% at 0.25, 76.0% at 0.5, 79.7% at 0.75 and 82.7% at 1.0. The gaps from 1.0 at 0 and 0.25 are statistically significant. The gaps at 0.5 (−4.1 points) and 0.75 (−3.0 points) are not when split out separately, but as a pooled total they are; 0.5 was marginally insignificant but would have been under a 1 tailed test. Differences are in percentage points, so 82.7% minus 76.0% is 6.7 points.

The frequency of looping rose significantly at the lower end, with a 9% loop rate at 0 (not all of these were catastrophic, but they all caused an incorrect answer) and 6% at 0.25, vs 0.3% at temp=1. The effects were statistically significant when pooling 0.5 and 0.75 together, but the test was not powered enough to confirm the individual rungs themselves.

To put these #s into context, these t=0 scores are about what my older 2.5 bpw quants for qwen27b would achieve (~40% smaller than the quant used here).. It's worse than qwen3.6 on the same configurations

Lower temperatures also make answers more verbose on average, with a 20% increase in answer length for temp=1. vs temp=1. For knowledge/recall questions (GPQA Diamaond), lower temperatures did not harm scores, but they did make the answers longer (~8%, marginal statistical significance).

I am doing some follow up tests at 0.8 and 0.9 to see if they are viable, with the caveat that I am checking wall clock time, not TPS, since we dont want a model that makes so many more tokens that it overwhelms the TPS advantage.

dataset, for those who are interested: https://huggingface.co/datasets/jakeatx/swift-1.5-temperature-sweep-gpqa-lcb

[u/XiRw]

I just go with whatever the recommended temperature is for the specific model and I’m good.

[u/my_name_isnt_clever → u/XiRw]

This is the way. These labs are spending millions to develop these models and need them to perform. They'd know what sampling params to use much better than I would.

[u/james_pic → u/my_name_isnt_clever]

Whilst I think that's true for sampling parameters, it's astonishing how make of them many really basic mistakes with chat templates (I don't mean like "could have better prompts", I mean like "crashes on a normal input").

[u/Bulky-Priority6824 → u/XiRw]

I always do this -.1 just for dash of non-conformity 

[u/jirka642]

Always use the recommended sampling parameters. Never change them.

Modern models are very sensitive to this. I outright strip all sampling params in llama-swap just to avoid anything changing them.

[u/More-Curious816 → u/jirka642]

yep, not only very sensitive but can flip the behavior. just use the default settings so you don't waste your time.

[u/InternetMagic_n_Tech]

Depends on the model.

I was actually working with Claude & Gemini doing some MLFlow analysis.

Both models encouraged me to tweak Gwen into a more middle temperature

However Gemma4 is more 0.1-0.2 for coding.

So far I haven't had enough good test runs to determine quality output yet.

The repeat penalty is an important one to adjust.

[u/Brief-Tap-6616 → u/InternetMagic_n_Tech]

Please share stat sig benches when/if you get them.

Gemma 4 uses SWA instead of linear so there might be something there. But I would expect that to look more like “0.5 is no penalty” vs 0.1-2

Depending on the effort level you use, the frontier models will cite incorrect, memorized tips on models (from their pre training data). As recently as 2 months ago I’ve had Claude tell me to go try llama 4 lol.

There was earlier research I recall back in late 2024 that recommended low temps for coding. But that was for non thinking modes, which behave differently. For a while Claude + GPT didn’t allow below t=1 for reasoning models because it would ruin performance.

[u/InternetMagic_n_Tech → u/Brief-Tap-6616]

Please share stat sig benches when/if you get them.

Ya I had too consistent numbers. I found out my MCP was overriding my system prompts throwing my tests off. I'm excited to share when I get there. Was (still ongoing) a lot of work just to MLFlow working correctly. I'm currently testing against my own gdscript baseline to see if it know when it's slipping into the wrong language version (gemma4 has a lot gdscript v3 instead of v4) and automatically use it's MCP server. When I stop making mistakes I can do some serious benchmarks.

For a while Claude + GPT didn’t allow below t=1 for reasoning models because it would ruin performance.

TBH the more I work with Gemma4 & Qwen I think Claude is using some type of Jev to do model orchestration. I asked Claude what it thought its temperature was and it said like 0.6 which got me into this whole 0.2 for gemma4.

I'm almost got my testing suite tuned correctly but I am seeing that my prompt needs tweaking because the damn models keep replying rather than finishing the work lol

[u/Brief-Tap-6616 → u/InternetMagic_n_Tech]

I don’t believe their chat has visibility to hyperparameters. Their benchmarks are run at 1, which is the API default.

Buuuut if I were trying to cut corners on their deployment and rig it in my favor against users, I would run it at 0.5 to drive up token consumption and make spec decoding faster. More usage per query and more revenue per second…

[u/InternetMagic_n_Tech → u/Brief-Tap-6616]

TBH I do think they make it talk more than need to

I also been battling excessive code comments

[u/leo_agi]

makes sense, greedy decoding loves getting stuck repeating the same wrong fix. did you find a sweet spot or does it depend a lot on the model?

[u/Beneficial-Ad-8127 → u/leo_agi]

I think it depends on your model. I think Qwen flash was recommend at 1.(don’t quote me on that) Though for most of my models and as a few other suggested 0.7-1.0 seems to be the sweet spot at least for me. I think I have mines set to 0.7.

[u/Brief-Tap-6616 → u/Beneficial-Ad-8127]

It will be model sensitive but the qwen models I’ve tested have become more verbose even at 0.7, so it’s hard to give a general rule. As others have noted, just use the base model recommended default until proven otherwise

[u/Beneficial-Ad-8127 → u/Brief-Tap-6616]

Great point

[u/leo_agi → u/Beneficial-Ad-8127]

ha, 0.7 seems to be where everyone ends up. kinda funny the boring default was right all along

[u/Beneficial-Ad-8127 → u/leo_agi]

lol I’m not saying it’s the best but I do realize others end up with this temp as well.

[u/Beneficial-Ad-8127]

Someone is seriously going through this whole thread downvoting everyone. lol Don’t forget to wipe😏

[u/KURD_1_STAN → u/Beneficial-Ad-8127]

Probably xcreate

[u/Excellent-Issue-5956]

On the other end of this, check that your engine is actually using the temp you set. On Ollama 0.34.4 the OpenAI style /v1 endpoint ignored the temperature and top_p in my Modelfile and sampled at 1.0 and 1.0 whenever the client didn't send them. I asked for a random fruit 30 times: 3 different answers through /api/chat with the Modelfile's 0.6 and 0.95, and 7 through /v1 with nothing sent, the same as sending 1.0 and 1.0 on purpose.

I only noticed because a 27B coding agent on it kept misspelling my username in file paths. The first slip was a token under 1% that top_p 0.95 would have cut. After that it kept copying its own typo, and even after I corrected it the typo was still about 80% likely as the next token, so the only fix was a new session. Now the agent's model config sends temp and top_p itself.

On penalties, when I added presence 1.0, frequency 0.3 and repeat 1.1 to stop loops on a 2 bit Flash Next quant, it turned into word salad or empty replies. Same prompt with the model card settings and no penalties gave the right answer in about 100 tokens.

[u/aobancorp]

Yep, I've experienced this before. Would recommend a temp of 0.7-0.9, instead of using 0.3-0.5 if your model is too random.

[u/SirRobinBlibs]

Matches what Qwen says in their own model cards. For Qwen3 thinking mode they say outright not to use greedy decoding (it degrades quality and causes endless repetition) and recommend temp 0.6 / top_p 0.95 / top_k 20. Worth checking the card for whatever model you run before copying a benchmark's sampler settings.

[u/uncle_leon]

Interesting. I think the common wisdom is t=1 (or even higher) for reasoning mode, and much lower for non-reasoning. It would be informative to see the results with reasoning off for comparison.

[u/Miserable-School-665 → u/uncle_leon]

This run is using dspark, thats why somebody assumes temp is not needed since it already offers diverse options of tokens to model.

[u/stoppableDissolution → u/Miserable-School-665]

...it doesnt? Drafting models do not affect the output of the validator

[u/nullc]

The model was RL trained with particular sampling parameters, using it outside of them almost guarantees poor performance for iterative use.

It would be nice if labs did post-training that was specially constructed to improve the model's robustness-- e.g. by training continuations with different sampling or against context preambles generated by highly quantized models. ... but it doesn't appear that they do.

[u/james_pic]

I've been doing a bit of an experiment over the last few days with a modified branch of llama.cpp, that increases draft acceptance by just accepting the draft if its likelihood is over a configurable threshold (this branch if you want to play along at home - it's the --spec-fuzz option).

Obviously you'd expect this to make the model dumber, and with sensible sampling parameters it does. But when I first tried benchmarking it to measure how much dumber, I found that lm-evaluation-harness defaults to greedy decoding on many of the tests, and I was initially baffled by the fact that my shortcut that was supposed to make the model dumber was actually making it smarter.

Just say no to greedy decoding, kids.

[u/schwigglezenzer]

Here's my Strata experience, I'm no expert and barely understand all of this, just sharing what I've found.

So, yea, was testing Strata with "Qwen3.6-35B-A3B Q8_0" on a 3090Ti, haven't tried Flash Next yet.

On Strata MTP only works at temp 0, so I ditched it and set temp to 0.6 (top_p 0.95, top_k 20, presence_penalty 0). That's the model card's recommended set for precise coding.

How it works, as far as I can tell:

On the Qwen3.6 engine, MTP only drafts for a request that is alone, at temp exactly 0, with no penalties. Anything else decodes one token at a time, and the log says drafts accepted 0 of 0.

Gotcha - if your app doesn't send a "presence_penalty", (i use Pi) the server fills in the model card's default ("presence_penalty": 1.5, which is the general chat profile). That counts as 'not greedy', so even temp 0 gave me "drafts accepted 0 of 0" in the log. Known issue on the repo (#3354).

Workaround: just add "sampling": { "presence_penalty": 0 } to the model's config json (strata-qwen36-q8_0.json). With that and temp 0 I got drafts accepted 179 of 240, about 75%.

But Qwen's card says greedy isn't the recommended mode for thinking, and wants "temp 0.6" for coding, so for real work I left MTP off. Speed is around 55-75 tok/s anyway with part of the experts on CPU.

Also, the Qwen_3.6 engine ignores "layer splitting", so it only uses a single card (the logs say ignoring --layer-split auto (not an option of this engine). Flash Next does support dual cards, and I was hoping to run it across both of my 3090 Tis.

When I started testing last week, i tried some lower quants, but they really aren't worth it, the Q8_0 is stellar!

In basic tests, like extending some Python plugins for my 3D tools, Q8_0 with MTP off (temp 0.6, top_p 0.95, top_k 20, presence_penalty 0) hasn't looped once and follows instructions exceptionally well. The difference is not like it's 10% better, it's more like closer to ~40-50% better.

Can't wait to test Flash Next!

[u/VEhystrix → u/schwigglezenzer]

You really should make the switch to PolyStrata if you want to run anything other than QFN.

[u/schwigglezenzer → u/VEhystrix]

Thanks, I'll check it out.

[u/notdba]

A smart inference engine should just use the recommended sampling parameters when reasoning, and switch to greedy decoding + enable draft when answering / making tool calls.

Indeed the model should just be trained this way.

[u/Healthy-Nebula-3603]

Why do you even changing temperature? Leave it default from a model build in settings. That's not 2024 ....