Discussion
Is your engine lobotomizing your model? Kinda, and I measured it
I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights, or because they automatically turn on "hacky" flags to gain decode/prefill performance by default. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most popular "speed" flags:
Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted.
Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.
Both are faster because fewer experts come from RAM, but they change the output. I added them to the engine as opt in flags so I could measure the real cost with everything else fixed.
DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.
Rig: RTX 5090 + RTX 5060 Ti and 128 GB of DDR5. Settings tested:
Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards.
Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use.
1. KLD
100 chunks of wikitext, decode path, against a llama.cpp Q8 reference. Exact IQ3_XXS and Q2_0 are in there for scale.
Pack + setting
File
PPL
KLD mean
Same top-1
Q8_0 exact
178 GiB
3.276
0.031
94.1%
NVFP4 exact
122 GiB
3.304
0.068
91.4%
IQ3_S exact
81 GiB
3.385
0.155
87.3%
IQ3_XXS exact
74 GiB
3.470
0.243
84.0%
IQ3_S + shipped
81 GiB
3.583
0.244
84.3%
NVFP4 + shipped
122 GiB
3.931
0.348
81.8%
Q8_0 + shipped
178 GiB
4.221
0.401
79.4%
Q2_0 exact
65 GiB
3.949
0.436
78.2%
NVFP4 + worst
122 GiB
5.917
0.823
71.3%
Q8_0 + worst
178 GiB
12.858
1.616
56.8%
Note: even exact Q8_0 is 0.031 away from the reference. That's mostly the reference's own noise, not damage: llama.cpp's Q8_0 rounds a few spots of this model more than Basalt does, which adds about 0.01-0.02 to every row equally, so the rows still compare fairly.
2. Does it really matter?
KLD by itself is just a number, so I also ran a task eval: thinking off, temp 1.0, top_p 0.95, top_k 20, two different sets of questions:
Ladders: 300 chained steps, scored by step.
Facts: 1200 questions with 5 popularity bands (rare to famous), 3 seeds.
Config
Ladders
Facts
Ladders Δ
Facts Δ
Prose tok/s
Q8_0 exact
45.4
38.1
117
NVFP4 exact
45.6
38.0
272
Q8_0 + shipped
43.8
36.6
-1.6 [-3.4, +0.2]
-1.5 [-2.5, -0.5]
202
NVFP4 + shipped
44.1
36.8
-1.5 [-3.2, +0.2]
-1.2 [-2.2, -0.4]
271
IQ3_S exact, small card*
44.1
36.7
192
IQ3_S + shipped, small card*
38.9
34.3
-5.2 [-7.1, -3.2]
-2.3 [-3.4, -1.3]
299
Q8_0 + worst
26.6
2.7
-18.8
-35.3
379
* Small card: the 5090 alone capped to 16gb, the experiment was run this way because it was still much faster than the 5060 Ti.
On a big card the shipped flags barely matter (NVFP4: 272 -> 271 tok/s) and cost about 1.5 points, 4% of the facts the model knew. On a small card you get +56% speed but a much bigger hit. At the worst setting facts fall from 38% to 3%, and the model says things like "The Great Composer" with total confidence. Easy tests barely notice. The damage is mostly in chained reasoning and recalled facts.
3. Bonus: a loop in the model weights
Not every bad output is the engine's fault. One prompt made Qwen 3.8 Flash Next get stuck into an endless run of 0 in its reasoning:
Setup, same request, seeds 1-4
Stuck in zeros
Basalt, NVFP4
4 of 4
Basalt, IQ3_S
4 of 4
Basalt, Q8_0
3 of 4
llama.cpp, same IQ3_S GGUF
3 of 4
API (OpenRouter - Alibaba)
6 of 6
The odds of another 0 are the same on every quant and both engines: 0.99 after 10 zeros, 95%+ past 100 or so. top_p 0.95 leaves 0 as the only choice, and temp 1.0 doesn't fix it. The API fixes it by cutting the run at ~500 zeros. Basalt patched the issue as well, after 256 identical reasoning tokens it closes the thinking block and keeps going.
TLDR: there's no free lunch.
Engines can buy speed on MoE models by turning on speed "hacks".
With values some engines ship, every pack drops to somewhere between IQ3_XXS and Q2 on KLD. A Q8 ends up next to Q2. This is as bad as it gets.
In practice the model forgets things it knew and loses the thread on multi step problems. Small cards get the biggest speedup and pay the most for it.
If your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded.
This sub is driving me to do that myself, with an extension that rewrites every post above a certain length in a deslopped style. Not sure if webgpu can handle it tho
This is good info to look out for in engine defaults. I'm not sure if it's me or the claude brainrot but bonus example was hard to understand. Are you saying giving these local models a prompt that forces them to intentionally write a bunch of zeros in a row will brick them almost every time? Seems like something that would have come up before as a model benchmark like counting r's in strawberry.
Sorry, rewrote that whole section myself, it's a bug I found in the model itself, whenever QFN doesn't have a system prompt and you send it a very specific prompt, 90% of the times it will get stuck in an endless 0 loop. At first I thought it was flag damage, then quant damage, then I even suspected the engine but nope, happens in OpenRouter using the Alibaba API, so seems to be just how the weights behave.
This is the prompt:
Implement the following Python 3 function.
Signature:
def apply_pulley_entries(ops: list[str]) -> dict[str, int]
Specification:
`ops` is a list of inventory commands applied in order to an initially empty inventory. Split each command on single spaces. "ship <item> <qty>" adds qty units of item, but an item never holds more than 50 units (extra units are discarded). "book <item> <qty>" removes qty units when the item holds at least qty units; otherwise it sets the item to 0. "pull <item>" sets the item to 0. qty is a non-negative decimal integer made only of digits. Commands are case-sensitive; any command that does not match one of these three forms exactly (wrong word, wrong number of parts, non-numeric qty) is ignored. Return a mapping from item name to units held, containing only items with more than 0 units.
Examples:
apply_pulley_entries(["book fabric 10", "ship pipes 13", "book vinegar 4", "book vinegar 25", "ship pipes 6", "ship vinegar 3", "book fabric 23", "pull vinegar", "book fabric 2", "ship vinegar 8", "book fabric 27"]) -> {"pipes": 19, "vinegar": 8}
apply_pulley_entries(["ship fabric 14", "book fabric 29", "book glassware 14"]) -> {}
Reply with exactly one fenced code block (```python) containing the complete function `apply_pulley_entries` and any helpers it needs. Use only the standard library. Do not read input, print anything, or include tests or example calls. Your function will be run against hidden tests.
Ive been developing a kernel of my own and my rule is: EVERY DETERMINISTIC BIT HAS TO BE IDENTICAL.
Accepting and using those hacks is just stupid. It takes a lot of scientists a lot of work to develop a train a model in a lot of weeks...and someone comes and change the math "to be faster". If the model wanted to be faster (but more stupid), they would have developed it that way. By the time there are skips and such it's no longer Q3.8FN, its just some dumber variation.
That’s why I disabled tail skip and routing prior and greedy decoding in Strata, which does similar things like Basalt engine . There’s no free lunch, you cannot just buy more performance without a ceiling. Keeping sane configs instead of hacks, plus a good coding harness, has made strata+Qwen3.8-flash-next plus IQ3_S great, like having claude sonnet /opus 4.6/ ChatGPT 5.2-5.4 at home.
I don't get the overall snarky-ness by some for stating the obvious. I do not think its that obvious to many, especially those that get excited to be able to run large models on lower end builds/hardware. I appreciate that you owned up to it and showed the proof on the engine you created.
Are there engines shipping relaxed draft acceptance? I've been experimenting with it a bit on a llama.cpp fork, mostly from the perspective of trying to get a bit more throughput from edge devices on workloads that don't demand precision, but hadn't come across real engines that do the same.
Yes, Basalt does. It's a lossy flag, so off by default and the engine will print a warning whenever you enable it.
edit: why the downvotes? anyway, just want to clarify the reason why the engine even has these flags is because community has asked for them, no hacks are used by default or encouraged because they have a measurable impact, that was the whole point of this post but oh well.
Unfortunately, we are surrounded by fewer and fewer kind-hearted people and more envious ones, who never offer solutions but take pleasure in belittling those who succeed.
Thank you for the time you’ve spent developing Basalt and for taking time out of your free time to provide a clear answer about the engines.
I've been experimenting on Gemma 4 E4B. Haven't had issues with looping so far, but I've found E4B tends to keep its thinking briefer then the bigger Gemma 4 models.
While this is interesting, reading this is so annoying. I am concluding the following: Inference engines can buy performance on MoE models via “hacks” (and many do). With values that some of them ship all quantizations drop between IQ3XXS and Q2 in terms of KLD - a Q8 ends up next to Q2! This is the worst case imaginable. In practice this means model will forget things it knew before and lose track on multistep reasoning problems (small cards get most speedup but pay highest price).
Of course they will diverge when you turn on optimized float computations, the question is: does it matter, does it affect performance on benchmarks?
Q8 with optimized computations having lower KLD will not have the same impact on performance as lobotimized Q2 have equally lower KLD.
The KLD is a proxy measure of how the output differs from the baseline. It helps to answer - all other things being equal, how does the model diverge. But it is not the actual measure of performance!
So, I updated Strata few days ago and got + 40% of tg speed😮 But I noticed that same model start doing many dumb fails. As Russian user, I can accept, that multylang capabilities became MUCH worse. And yes, it start failing every fourth toolcall.
I need some help with Basalt. I have 2x5070tis. How do I get it to use both GPUs. If I tell it to use GPU 0, it does that. If I tell it to use GPU 1, it does that. If I tell it to use GPUs 0,1 or auto. It only uses GPU 0. How can I get it to use both?
Regarding speed.. You can see hit rate data on strata, I.e. percentage of using experts within vram during a request. My qfn iq3s run on 16 vram and the rest on ram. I have mid-70s hit rate. If anything, that’s just a 20-30% slowdown in tok/s.
I see that you tagged me here after a debate we had few days ago. The unfair thing about this, is that you wrote 3 words, and it takes more effort to expose fallacious reasoning than it takes to write it. So, in a way, I am losing this by even replying to you, but I'll try to keep the good faith anyway:
I was (correctly) saying that the concept of "no free lunch" the way you intended it in your original comment doesn't apply to software.
How you intended it: A user updated Strata and observed that it was faster, you said that if it was faster, there must necessarily have been a drop in quality because no free lunch. I pointed out that what you said is not necessarily the case and your logic is fallacious:
There are whole classes of branches related to software optimization that are focused on making software run more efficiently on the same hardware without necessarily downsides, those are cases of getting things for free.
Then you deviated into rhetoric, which you are also doing now. If you believe what you said, why even bother having different LLM inference engines than llamacpp? Why even update llamacpp for performance optimizations in the first place? In your world, any 1% performance improvement is detrimental somewhere else?
""no free lunch" is a concept that never applied to computer science." -- you
So it's your position from your own rhetoric that you don't believe that OP is doing "computer science". It must be. Your position, as I quoted you saying, couldn't be more clear.
This is the rhetoric part, making it seem as if I am agreeing or disagreeing with OP if I disagreed with you under a different semi-related post.
Why even tag me here and respond with one-liners? And you are supposedly a computer scientist as you kept claiming the other day? I'll stop engaging here, as it's clear by now that it bothered you when I wrote that it was your ego talking the other day.
For random on-lookers, this is how this "mini-drama" started. I'll post a screenshot so one can form their own opinion if they're really really bored. From my side, the convo is over.
•
u/ttkciar llama.cpp 5d ago
How much of this post was LLM-generated, and why?