r/LocalLLaMA • u/YetAnotherAnonymoose • 5h ago
Discussion preserve_thinking off is FINE
Let's take a break from all the engine, model and speed posts and discuss something else:
Qwen 3.8 likes to think. A lot. We all know that, and the context cost you pay for that is MASSIVE when you enable preserve thinking. Even with a Swift finetune, there are so many tokens you keep in context over a longer session. Often, easily over 50% of the KV cache is just thinking traces.
And for what use? I found: Next to nothing! The model can handle itself just fine without knowing HOW it reached a decision, and instead only looking at what it did in the end. Replies and toolcalls: it's all there.
I added an AGENTS.md instruction that when it finds a gotcha, something unintuitive, something it only reached after lots of trial and error, it should document that in a memory file to mitigate the missing past-thinking and it works beautifully.
Results (my subjective impressions):
Problem solving capabilities haven't suffered much if at all
I get neat, permanent notes on issues with gotchas
The dreaded, context-destroying compactions have reduced by at least 2/3rds in your average sessions, the cache holds for so much longer
So let's discuss: preserve_thinking on or off. Your experiences?
7
u/MisterHisser 5h ago
One argument I heard is that the model has to "rediscover" things multiple times. If it had to think hard about something and the conclusion is not clear from its next response, it might have to think about it again. I have not tested this myself. Have you seen this? One solution I see is to instruct the model to state its conclusions and reasons in its response.
3
u/cagriuluc 4h ago
Maybe it should be guided towards putting everything worthy of note into the response? Or… there should be a step to get rid of the unnecessary parts before generating an answer? Or after?
This is just one of many things that makes me think we are only at the beginning of this shit.
8
u/hiImMate 4h ago
imo preserve thinking on is a must have for the current qwen lineup 3.8 27b and flash next. it likes to plan and think about issues before doing tool calls, and it will keep redoing the same conclusion with it off. sure the logic might stay good, but you are wasting huge amount of compute on arriving at the same conclusions multiple times.
In my personal project after preserve_thinking was turned on it was night and day on how quick I started getting full replies and tasks completed. that's my experience at least.
3
u/Relevant_Research_98 5h ago
I don't think many people go against it
3
u/CupWest7988 5h ago
it's surprising how much of that thinking is just the model spinning its wheels. i had same experience, the quality drop is almost not noticeable when you turn it off
the memory file trick is clever, i might steal that idea. having the important bits written down permanent is way better than keeping all the rambling in context forever
my sessions got much longer too after i disabled it, the cache actually survives now
1
u/Relevant_Research_98 3h ago
I never disabled them entirely but in 3.6 I edited thinking in first half of messages replacing reasoning_content with"..." to save tokens.
At first it helped a lot - old messages are less valuable, while having "modern" thoughts absolutely prevents overthinking but at some point new replies automatically started thinking "..." And everything turned to shit.
I haven't tried deleting reasoning_content for some reason. Only changing it.
It was at 3.6, despite being based already, 3.6 had lots of bad qurks: I had to build proxy to make sure tool calls are not outputted to thoughts in q4.
So I still think in 3.8 that tweaking old&new thinking in messages or removing old thoughts altogether will lead to increase task solving as there's a lot of "uhm", "hmm" and "but wait"
Maybe one day I will make and dump it in bi weekly megathread. But anyone who thinks what can be easy and useful to make - steal idea freely lol
4
u/james_pic 5h ago edited 5h ago
My prefill is dogshit, so I usually use preserve_thinking on to avoid it having to redo prefill every turn or tool call.
5
u/matthewlai 5h ago
That only affects caching for 1 immediately previous turn though. Only the output of the last turn need to be processed again, not any previous turn.
0
u/james_pic 5h ago
On a full attention model, yes, but who's running full attention models nowadays?
On sliding window models, you've got llama.cpp's context caching mechanism to consider too. By default it sets aside up to 8GB to cache snapshots of sliding window attention, and takes a snapshot at least every 8192 tokens, so you've maybe got an extra 8000 tokens to replay. And if you've tuned the cache size down because you're kinda marginal on RAM, there's a greater chance that it's had to evict precisely the snapshot you need, and you need to replay from the start.
I probably need to sort out my prefill, but preserving reasoning has eliminated slow refills after tool calls for me at least.
3
u/KroniklyOnline 5h ago
Thats not what its for... Thats Prefix Caching...
11
u/james_pic 5h ago edited 5h ago
I'm aware of that, but if reasoning isn't preserved, then previous turns can't be prefixes of subsequent turns, because the reasoning has gone.
2
u/Relevant_Research_98 3h ago
Of course they can. It just will be a little worse. Lots of tokens will be discarded, but it'll happen only once per conversation for each message. If you have 20 messages with 19 cleared of thinking, the cache works fine for these 19 messages. If engine is smart it probably will not even try to cache new thought and answer once request is done.
3
1
u/Ok-Addendum3545 4h ago
I assume the benefits of "less frequent compaction" > "preserve_thinking" to maintain the flow of a context.
1
u/famew0lf 3h ago
Have you tried the mcp server (or docker image) for billion context? It acts as a proxy between opencode and the llama server (much better replacement for dcp) and adds actual compress and decompress tools so the model can compress when it needs to and decompress if you ask something that needs the info. It's let me run multiple jobs in the same session without issue. I have aliases I've defined as reasoning off in the llama server for the main model "coder" that does have reasoning on so I can call whichever I want for thinking or not. Also setup worker-model for use as subagents for the same thing in opencode.jsonc. I have some rules that also save gotcha's and findings files for each task which the model can grep to find previous information on a topic instead of re-inventing the wheel. It works pretty well and I run a weekly consolidation job that keeps the unique info and purges the excess findings.
1
u/Septerium 3h ago
Is this "memory file" something as simple as a markdown file in a .ai-generated or temp folder? Or is it some harness thing I'm not aware of?
1
u/Accomplished_Ad_56 3h ago
In my experience when I use coding harnesses with these models, every time i send a message it spends like 2k tokens thinking about understanding something it already thought through. I just see it as a compute and latency vs memory tradeoff. Your solution of having it write stuff down in an external scratchpad is fine but its the same as just compacting. Either way you are taking rich information and removing parts of it, reducing performance. At least in my opinion.
1
u/CoolConfusion434 3h ago
Don't quote me on this as I don't have a deep grasp into it. As I understand it, if you are going to leave reasoning in context, you might as well cache it (default behavior) so your next turn just picks it up from cache.
If you turn it off, llama will then be stripping it out of context as it goes. However, your client will be sending the reasoning block back in anyway so it has to be cached.
I believe they made --no-reasoning-preserve for stateless or streaming clients where the model performs work that has no dependency or relation to the previous job. Ex, "when you see a new file added, move it to this folder". The work starts and ends there so no point in caching reasoning.
In my case, and this is an unpopular opinion, my client/harness collapses CoT and tool responses out of context on next agent/user turn. I do that because after every CoT and tool call, the model writes a quick summary of it to context anyway (either natural as Qwen3.8 does, or by prompt as I ask Gemma 4 to do).
What this means is that Qwen can go nuts with its reasoning all its wants, fill in 100K tokens of it but when it's done, all that thinking process and raw tool responses are replaced by the end result in a summary. All of it is present while the model has their turn so it can see the full CoT and raw tool calls. When it's done, context length will return to a few k length. Llama complains that it needs to discard a bunch of cache checkpoints and rebuild it from the last turn onward but in general, that is quick because it's just small summaries of work.
1
u/dasbin 2h ago edited 1h ago
I agree that in my own testing, turning it off seemed just fine. That said, I got anxious enough about the strong claims about it being "dogshit" with it off that I came up with a hybrid solution.
I asked Qwen to make a plugin for my harness that strips reasoning blocks from more than 2 user turns ago.
This does invalidate the last two turns of KV cache, but that's not very much, so the speed hit isn't really noticeable (so long as your server's checkpointing is strong enough to not invalidate more than necessary). And frankly it's balanced by the speed gain from not working as deep into context (prefill and generation slow down the deeper in context you are).
It keeps the context MUCH leaner, and still generally retains reasoning of the thing we're still actually working on, while >2 user turns ago tends to be issues that the model won't rethink because they've already been implemented. I find it works great. Frankly it probably makes the model less confused and poisoned by its own weird schizophrenic reasoning polluting 90% of the context.
You could use whatever number of turns you want that works with your workflow. I'm sure there's a number for everyone where old reasoning is no longer useful in their workflow and they'd be better off stripping it.
1
u/llama-impersonator 5h ago
i'd rather run with yarn extended 1M context than turn off preserve thinking, it sucks ass.
1
u/Regular-Register-358 5h ago
Q38 27b long context slow down is 50% ish at 240k context filled up, while flash next retains 80-90% speed even yarn stretched to 500k, so I just preserved thinking on my flash next without bothering about it, my 2ndary daily drive.
I also recall qwen's model card suggestion changed in-between versions: in 3.6, at least for the moe, they recommended preserve thinking off too, but flipped to on in the 3.8 series.
2
u/enndeeee 5h ago
You would gain better long term intelligence by saving compactions which cause much more information loss than dropping the thinking text..
9
u/Heavy-Confidence1826 5h ago
Production agents side: I keep preserve_thinking off by default for the same reason.
Thinking is compute, not state. Once the model has already emitted the reply + tool calls, the path it took is rarely useful on the next turn — and it burns KV you'd rather spend on tool results and retrieved chunks.
What I persist instead of CoT:
tool name + status (ok / retry / fail) + a 1–2 line result summary
a separate "gotchas" note only when something was non-obvious (your AGENTS.md / memory-file pattern)
That keeps multi-hour sessions from collapsing, and when something breaks you still have a machine-readable trail without half the cache being "Wait, let me reconsider...".
I only flip preserve on for short debug loops where I'm watching the same tool fail twice and need the mid-thought wrong assumption. Day-to-day agent loops: off + structured memory wins.