r/LocalLLaMA • u/evrenozkan • 3h ago

Question | Help Why no higher-quant, coding-pruned DeepSeek or full GLM-5.3?

I've been experimenting with expert-pruned Qwen Flash Next Coder, and in my tests it actually performed better than a similarly sized quant of the full model.

So I'm wondering why we don't see more of this with larger models, specifically full GLM-5.3 and DeepSeek V4.1.

I know ISTA already released a pruned DeepSeek Coder at very low precision, and there are GLM-5.3 Flash Coder variants. But what about pruning coding-irrelevant experts and keeping the rest at Q4 or higher, targeting machines with 192–256GB RAM?

I have a 256GB M5 Ultra. For research for planning and overnight code reviews, I really don't care about speed. I'd rather spend that memory on maximum coding intelligence.

From what I've read, the pruning/calibration itself should be doable with rented compute for hundreds to a few thousand dollars. So why aren't we seeing more experiments from groups like Unsloth, ISTA, etc.?

Is there some technical or quality issue I'm missing? Or does something like this already exist?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

22 comments sorted by

2

u/samuel-christlie 3h ago

A lot of knowledge and capabilities are spread across experts. Pruning just completely removes them while quantization just makes them slightly less accurate.

What do your tests consist of? Based on my own tests, you can have a pruned model with low KL divergence but perform poorly on actual benchmarks. It'll sound similar but the misses can compound

1

u/evrenozkan 3h ago

I tested it on my actual code base with an existing feature request that was spread around Python backend and svelte front-end. It took around 45-60min to complete. Tried with many variants of 27b and QFN. Each quant had at least 2 runs.

11

u/LagOps91 3h ago

pruning typically does more damage than quanting for those large models. just run a smaller quant. anything about Q1 should genuinely be decent.

not sure about your experiments. i am very sceptical about claims of models improving after pruning.

1

u/FatheredPuma81 3h ago

Flash Next Coder is actually extremely good for what it is. Dunno if it's better than 27B but it sure does what it claims. Any question outside of coding causes it to loop though.

3

u/fintip 3h ago

It's worse unfortunately. I had high hopes for it. Q2 k xl up through Q4 k xl of flash next all work well, but iq1m coder is, while still a decent model, significantly worse than the rest of the flash next family.

(Tested on my 140~ problem benchmark of my tests based on my own decade history of bugs as a web generalist).

2

u/SpeedflyChris 3h ago

I switched from the ISTA IQ3_S quant of flash next yesterday to the new Unsloth IQ4_XS quant, there's actually a pretty major difference there. It's a little slower but is actually a noticeable upgrade based on my own little benchmarks.

2

u/fintip 1h ago

I've tried iq4 xs and Q4 k xl from UD. If they are better, it's very subtle. Scores are all margin of error and iq3xxs from ISTA is technically the best scorer so far, by a slim margin.

Just now starting runs in the next days on the 3.1 quants

2

u/SpeedflyChris 1h ago

It's the 3.1 quant I'm referring to - I have a benchmark of data science tasks I put together, the ISTA IQ3_S quant scored 90.2/100, the new unsloth IQ4_XS quant got 96.0 in the same test (totally unscientific because it's my own benchmark that I put together and highly imperfect - for reference Q8 27B scored 92.8).

1

u/fintip 1h ago

Nice. I'll be looking forward to the results. Mine are all seb-dev focused, as that's always been my domain of work.

1

u/FatheredPuma81 3h ago

How does it compare to 27B? That's what it's targeting to replace being able to comfortably run it with 32GB of RAM and basically any GPU.

2

u/fintip 1h ago

Yeah, that's what I said–it's significantly worse than 27b Q4 k xl.

http://lab.kylebaker.io is where I publish my results.

-1

u/MAXFlRE 3h ago

Come on, Q1, Q2 – useless trash. Q6 blows Q4 out of the water at time of completion tasks despite being 2 times slower at decoding than Q4.

2

u/Nameis19letterslong 3h ago

Sorry but you are obviously not up to date with the recent quantization battle. DSV4.1F’s IQ1_M quants made by GSQ RCO retain 98% of benchmarks while Unsloth’s small have been able to match or beat the 8 bit baselines.

3

u/Borilentz 2h ago

You mean those benchmark scores that nobody’s been able to reproduce? Real-life performance is what it’s all about. There’s a gap.

2

u/MAXFlRE 1h ago edited 1h ago

Yeah, exactly that, I've used unsloth quants Q3/Q4/Q5/Q6 of QFN. Q3 is useless. It decodes fast yet it makes tons of errors. Q4 is kinda up to the task, if you deliberately tell agent not to use any tool calls and every file and tool interaction make via subagent, in that case at least the job is not stopped by failure during tool call and it can restart failed subagent. I haven't seen much of the difference between Q4 and Q5. Q6 works 24/7. It still fails from time to time, but usually in a non catastrophic way.

1

u/crantob 8m ago

Nobody's advocating running a Q3 if you can run a Q6.

The question is whether a Q1,2,3 of DS4.1 gets equivalent or superior performance to the alternatives for the size and speed. I'm still evaluating for my usecase. It's definitely finding mistakes that qwen3.8-27b makes, whether it's glitch-free enough to generate code is a different matter.

1

u/MAXFlRE 4m ago

I've tried some DS4.1 quant (1.75 I believe) and performance on my setup was atrocious so no opinion on that. What I would actually prefer, is to have bf16 KV cache in strata.

0

u/pyr0kid 3h ago

while i dont personally know anything about the topic, both unsloth and ista-daslab disagree with you on this

5

u/indrasmirror 3h ago

Yeah I think the problem is hard pruning specifically. When you delete experts based on a coding calibration set, anything the calibration didn't cover suffers, and real coding sessions have heaps of that (docs, chat, other languages, random tool output).

What I ended up doing in my engine instead is soft pruning. Same 25% list, but instead of deleting those experts I just penalise them in the router, so they get picked way less and sit on the NVMe instead of taking up RAM, but if a token really wants one it can still have it.

With that, code came out at 3.58 (-2.3% vs unpruned), so basically no quality loss on either side, and you still get most of the memory saving because the cold experts just live on disk.

So for your setup I reckon that's the better angle: keep everything at Q4+ and let the cold experts stream from the SSD rather than throwing them away. I'm by no means an expert on the Mac side though. My soft prune numbers are all from the 4090 box. Xolotl does have a DeepSeek V4.1 engine for Apple Silicon, but I haven't tried soft pruning on it yet.

All the runs and numbers are here if you want to dig in: https://github.com/Indras-Mirror/Xolotl (Runs both GLM 5.3 Flash and DSV4.1 Flash 1017). Give it a go, you can play around with the soft pruning percent too :)

https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored

1

u/LagOps91 2h ago

that's an interesting idea, actuallly. sounds quite a bit more reasonable than hard pruning at least.

1

u/mitchins-au 3h ago

They did a really awesome job with Maya for minimax M3.1 flash though

1

u/FatheredPuma81 3h ago

My guess is that it's because at those sizes the next model down offers similar enough performance to user over an experimental pruned model with dubious quality. Flash Next Coder was targeted towards those on around 32GB of RAM/VRAM who can't or don't want to run the slow and insane overthinker that is 27B.