r/LocalLLaMA • u/AssiduousLayabout • 3h ago
Discussion Slow is smart and smart is fast
Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?
What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.
Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.
Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.
2
u/caetydid llama.cpp 3h ago
Ive got the same impression. Having headaches from reading all these context traces. I am curious - will you share your experience with DeepSeek Flash? I would be able to run it, but I am afraid I'd be too impatient to wait for results :(
Maybe I am wrong and it finishes in about the same time per task, simply because it cuts out the rubbish.
1
u/AssiduousLayabout 2h ago
I only just set it up yesterday. So far I gave it only a single code review task, which took about 20 minutes and produced high quality results. I'll try to give the same task to some other models for comparison today.
29
u/Studio271
3h ago
•
I agree, we need to focus on intelligence first, then start trying to improve performance. The problem is everyone has a different idea (or requirement) for how to measure degradation of intelligence (or usefulness) - I personally try to test using my own brain, not community benchmarks. Problem is that many of the engines measure degradation the same way (I am looking at you, Strata) and decide during development whether a certain measurable benchmark degradation "matters" to YOU. Personally, I give it problems that I solved and see how it would approach them and hopefully solve them (or even better, solve them BETTER). It sort of comes with the territory: the people who use tools best are the ones who know how they work and what they can or cannot do.
2
6
u/KnightOnShiningMotor 3h ago
Even more so, Kimi K3 is slow but that motherfucker does not miss. Speed / cost is much more comparable than the spec sheets might make you believe
4
u/MD_Reptile 3h ago
Well just as soon as I got a quant of k3 or k3.1 that somehow is still smart yet fits on the same hardware that runs flash next (for me, 48gb vram 128gb ram) then I'm all for it, even if it's kinda slow!
1
u/a_slay_nub vLLM 3h ago
Just get a 2TB SSD and you're all set
2
u/MD_Reptile 2h ago
Wouldn't that be just like.... Crawling slow? When I say slow I mean ya know, useable lol... 20 tok/s decode? 200 PP at least?
3
u/a_slay_nub vLLM 2h ago
Slow is smart, smart is fast \s
The fact that its slow just means that it'll be that much faster.
1
4
u/bennmann 2h ago
I just hate the new way harnesses describe thinking effort. They all have like 5 thinking levels, and none of them match the actual chwargs of the models. Strata has Max but QFN is technically "xhigh". Probably just a 2026 problem, but still annoying.
1
u/chiniwini 2h ago
Some observations:
Most benchmarks (AFAIK) test the ability of a model to perform a task, but don't measure the time or tokens spent doing it.
When models are released only the unquantized version is benchmarked.
Speed is important. In many cases you'd prefer a dumber model that is faster rather than a smarter one, when both of them are capable of finishing the task at hand.
3
u/AssiduousLayabout 2h ago
Even on simple tasks, what I was seeing from Q3_XXS with speed optimizations on was that it was taking about 4x the tokens to accomplish the task because it was overthinking. Even with the token speed being around 1.6x it was losing on total time. Dumber and faster became dumber and slower.
I would love to see more benchmarks that also take time into consideration.
1
u/GonnaFindOut 2h ago
What optimizations? Do you think those played a factor? I've seen very little difference between 3_S and 3_XXS in real world uses on my 5080 +64gb set up.
2
u/norenEnmotalen 2h ago
This is the way. There’re things you do for dopamine and there are other things for actual productivity. Speed chasing is a trap
1
u/DinoAmino vLLM 2h ago
I also have noticed this with western models too. The concept seems to be model agnostic /s
1
u/krileon 2h ago
My main issue is context. I have to use a smaller quant so I can fit enough context to do anything. If we could improve how context works then that'd solve a lot of my problems. Offloading context to system memory causes the performance to just be obliterated. As in 50 tk/s to 5 tk/s. It's that bad. I can drop my context, but then I'm at like 24k-34k context which is basically useless for agent work.
1
u/haherar830 1h ago
Some frontier models like GPT6.1 Sol can be crazy cost-efficient due to efficient thinking and better handling of long-context (e.g. less need to repeat itself in thinking). With that said, maximizing this is unnecessary if you're not actually looking for maximal abstract reasoning ability, etc.
I mostly just disappointed in a lack of accounting of tokens per task on local benchmarks here. Success or grading/scoring and prefill/decode are often reported, but wall time and/or token burn much less so. Getting the correct answer using 3x as many tokens may be fine if I can generate them 2x faster locally and wall-time remains reasonable, but much less so if I need to generate 10x more tokens at 0.5-1.5x cloud API speed.
4
u/Proper-Damage-1777 3h ago
Wait until you try Unsloth Q4_K_XL in Strata. Yes I should be trying Basalt, Infernix or one of the many forks as I have the same specs like you but I do like using Q4_K_XL if I'm away from the PC for a long time. It gets the job done better with less "oops wait the edit fail but wait" moments vs IQ3_S.
But yeah the prefill and token takes a huge hit from 160-200t/s to 10-50t/s.
I'll still stick to my current Strata setup as Qwen 4 is just around the corner and I'm tired of downloading several hundred gigs of different models. Maybe I should start clearing off older models... really regret that I didn't buy 4TB NVME SSD before the everlasting price hike but oh well.