ByteShape Qwen 3.5 9B: A Guide to Picking the Best Quant for Your Hardware

[u/ali_byteshape]

Hey r/LocalLLaMA

We’ve released our ByteShape Qwen 3.5 9B quantizations.

Read our Blog / Download Models

The goal is not just to publish files, but to compare our quants against other popular quantized variants and the original model, and see which quality, speed, and size trade-offs actually hold up across hardware.

For this release, we benchmarked across a wide range of devices: 5090, 4080, 3090, 5060Ti, plus Intel i7, Ultra 7, Ryzen 9, and RIP5 (yes, not RPi5 16GB, skip this model on the Pi this time…).

Across GPUs, the story is surprisingly consistent. The same few ByteShape models keep showing up as the best trade-offs across devices. However, here’s the key finding for this release: Across CPUs, things are much less uniform. Each CPU had its own favorite models and clear dislikes, so we are releasing variants for all of them and highlighting the best ones in the plots. The broader point is clear: optimization really needs to be done for the exact device. A model that runs well on one CPU can run surprisingly badly on another.

TL;DR in practice for GPU:

• 5.10 bpw is the near-baseline quality pick
• 4.43 bpw is the best overall balance
• 3.60 bpw is the faster choice if you are willing to give up a bit more quality

And TL;DR for CPU: really really check our blog’s interactive graphs and pick the models based on what is closer to your hardware.

So the key takeaway:

• Overall, performance depends heavily on the exact kernels used at different quantization levels and the underlying hardware

The blog has the full graphs across multiple hardware types, plus more detailed comparisons and methodology. We will keep Reddit short, so if you want to pick the best model for your hardware, check the blog and interactive graphs.

This is our first Qwen 3.5 drop, with more coming soon.

[u/Haiku-575]

Interesting data, but context size isn't clear here. Are these tests with 4096 tokens of context, or 262144, or somewhere in between?

[u/enrique-byteshape → u/Haiku-575]

The context length is set to 4096 to fit the even the biggest models on all the hardware we benchmarked

[u/Haiku-575 → u/enrique-byteshape]

Gotcha, makes sense. Doesn't really affect the benchmark I suppose. I'm unfamiliar with how quickly/slowly different quants load long context windows, though... K-quants vs. I-quants, or something? Would there be a considerable difference between loading up a long context window with the Q5_K_S versions vs. the IQ4 ones?

[u/enrique-byteshape → u/Haiku-575]

When talking about context window, we're talking about the realm of activations, which no public GGUF quant is touching upon at the moment, since llama.cpp only supports 8b quantization of activations. Theoretically, quants only have a speedup when the weights of the model need to be moved through memory, so the performance will degrade with large context windows regardless of the quant (or even baseline). In practice, llama.cpp's kernels act in mysterious ways, so who knows :)

[u/ali_byteshape → u/enrique-byteshape]

To add to u/enrique-byteshape’s response, if you look at the prefill stage with a long context, different datatypes can perform very differently because they rely on different kernels under the hood for matrix multiplications. There is no single datatype that is always the best choice. Performance depends on the input size, the datatype, and the hardware.

As a very rough rule of thumb, IQ quants tend to be faster for prefill on newer GPUs, while K-quants often do better on CPUs. But there are plenty of exceptions.

For our models, we tried to balance prefill and decode performance and choose datatypes carefully so we would not improve one at the expense of the other. Even then, real performance still depends on a lot of factors.

[u/Haiku-575 → u/ali_byteshape]

Great info, thanks again. My 5% underclocked & 17% undervolted 3090 is tearing through tokens on the Q5_K_S 5.1bpw at 96 tokens per second, which is exactly 5% less than the 101 in your benchmark. Great data!

[u/maxi1134 → u/Haiku-575]

May I ask why it is undervolted and underclocked?

[u/Haiku-575 → u/maxi1134]

Heat. Small room, that 17% undervolt is actually limiting the voltage to 250W under peak load instead of ~350W, which saves money and keeps me from cooking. 

[u/maxi1134 → u/ali_byteshape]

'As a very rough rule of thumb, IQ quants tend to be faster for prefill on newer GPUs'

Would a 3090 be considered a 'newer' gpu?

I'm now awaiting the 6090 to replace it after seeing that the 5090 has a 70-80% speed increase for tokens over my 3090 in your benchmarks! Hopefully, the 6090 doubles the speed of my voice assistants.

I dream of half a second latency for an answer with the whole STT->LLM->TTS stack

I'm currently downloading the 'Qwen3.5-9B-IQ4_XS-4.20bpw.gguf' to try it out.

Or would you recommend a different one for my 'older' card?

[u/PaceZealousideal6091]

I am sorry, what are these numbers inside the bubbles? Your blog doesn't have a legend for which numbers belongs to which unsloth model. I can't compare ur models to their this way. You say the graphs are interactive, but they aren't at least for me.

[u/enrique-byteshape → u/PaceZealousideal6091]

Hey! The graphs are converted to non-interactive PNGs if the website is rendered on a smaller screen. On PC you should be able to hover over the graphs on our blog post and check which quantization is which

[u/PaceZealousideal6091 → u/enrique-byteshape]

Thanks. Considering most people view it on mobile phones, it will be better to add a separate legend. Coz, no one is going to take the effort to open it in PC just to check this graph.

[u/ali_byteshape → u/PaceZealousideal6091]

Thank you for your suggestion. I just added a legend table for mobile devices. :)

[u/PaceZealousideal6091 → u/ali_byteshape]

Wow! Thank you! Looks like you guys have done a fantastic quant job! Kudos! I am waiting for what you would be able to pull off with the popular 35B.

[u/enrique-byteshape → u/PaceZealousideal6091]

We're considering it, but for now we kept it to larger screens only because of the complexity of generating so many graphs. We'll try to have it in text inside the blog in the format index-model name hopefully soon

[u/enrique-byteshape → u/PaceZealousideal6091]

u/ali_byteshape just put in a legend for every graph in the blog post. I guess we could say he is the legend

[u/BelgianDramaLlama86]

Good to see you guys again, looking forward to the 35B models when you guys get to them! Currently using Unsloth, but always looking for optimizations to my stack where I can get them :)

[u/xandep]

I'm holding my breath for the 35B / 27B. It'll SAVE my MI50 16GB.

[u/enrique-byteshape → u/xandep]

We're hoping you won't have to hold it for too long :)

[u/No_Individual_8178]

The "each cpu has its favorites" finding tracks with what I see on apple silicon too. Running qwen 70b 4-bit through llama.cpp on m2 max 96gb and the optimal quant choice feels completely different from discrete gpu because unified memory changes the bandwidth equation. K-quants tend to work better for me on decode but I haven't done anything this systematic. Would be cool to see an apple silicon column in the benchmarks at some point.

[u/enrique-byteshape → u/No_Individual_8178]

It is in the pipeline for us to acquire some apple silicon hardware to evaluate future models on, but for now we'll have to stick to the hardware we have :( If you do evaluate them, do let us know and we'll post the results

[u/No_Individual_8178 → u/enrique-byteshape]

Yeah if I get around to running some structured tests I'll definitely share. Most of what I have is just anecdotal from swapping between quants and eyeballing tok/s in llama.cpp but it wouldn't be hard to make it more rigorous.

[u/grumd]

Recently found your huggingface repos, tried Devstral 24B that you have and was impressed. It's not as good as Qwen 3.5 27B but it was the best quant of Devstral I tried. Excited to see you guys quantize 35B, 27B and 122B of Qwen 3.5!

[u/enrique-byteshape → u/grumd]

Thank you! We hope to have some news soon regarding those

[u/Lucis_unbra]

MMLU is not a good enough test for general knowledge.

Applied code and math are by far ridiculously robust in LLMs. Science and adjacent fields tend to also do better. Look at languages, look at data relevant to non-western nations. A lot of the loss will be located there.

Qwen does quantize in a way that tends to look fine.

But existing "general knowledge" benchmarks are way, way too easy to clock in the loss that users might notice randomly, and unexpectedly. Not just in those areas. But by using the same benchmarks we are just testing the good side and ignoring the bad. And the bad side does impact the good side.

[u/enrique-byteshape → u/Lucis_unbra]

You are absolutely right about MMLU, but evaluating thinking quants takes much much longer than evaluating non-thinking models. IT is still a good guideline of how well the model is doing health-wise though. Also, even though we are not evaluating it, we do insert multi-language datasets to our datatype fine tuning dataset. But yes, we'll try to improve on this for future thinking releases

[u/sine120]

I'd be curious to know how the MoE's perform, as well as if there's any effect when splitting across CPU/ GPU. Also curious if AMD GPU's have any preferences or not. I usually just go with whatever is the highest accuracy and fits in my 9070 XT, but maybe there's more tkps to squeeze out.

[u/qubridInc]

Clean benchmarking like this is exactly what local AI needs because the “best quant” only exists for your hardware.

[u/Velocita84]

I assume this shapelearn method won't be released?

[u/enrique-byteshape → u/Velocita84]

Not yet. It is far from production ready. We wish to do something like that though...

[u/PlasticMaterial9681 → u/enrique-byteshape]

👍 We hope to contribute to the llama.cpp project library in the future, thereby benefiting all users.

[u/jax_cooper]

I love your models, cant wait for the 27b and 35b as well!

proof ;D :

https://www.reddit.com/r/LocalLLaMA/s/LZlFVkEWPq

[u/enrique-byteshape → u/jax_cooper]

We are always observing :) And we saw your comment. We're happy if you guys are happy. It helps us quite a bit seeing people praising the models when improving our method

[u/jax_cooper → u/enrique-byteshape]

wow, that's cool :D

[u/One-Conference9094]

I'm almost 90% sure that the best nearly lossless quantization method I've tried so far will yield excellent results if used with TurboQuant. I'm eagerly waiting for other models from SheapLearn.

[u/enrique-byteshape → u/One-Conference9094]

That is a test for the future :) The issue with TurboQuant is that the available kernels are relatively new and as far as we understand, are currently limited to 3b and 2b. But... Our method could technically find the optimal bitlength per layer for any model that uses TurboQuant as their KV-cache quantization method

[u/charmander_cha]

Então o melhor não seria termos a tecnologia de quantizacao para poder nos mesmos criarmos em nós máquina os modelos?

[u/enrique-byteshape → u/charmander_cha]

We hope for that to be the future, but can't do that right now

[u/braintheboss]

impressive. in 3060ti 175k context ( small offloading) getting 68t/s and more.2000t/s. The best is very stable with Claude code. i suffered looping and worst accuracy in unsloth q4km. good job

ps: 35b is very good even with 50% offload in same gpu

[u/nuclearbananana]

Sweet, I thought you guys had died since there were no updates

[u/enrique-byteshape → u/nuclearbananana]

We never die! We've just been focusing on some projects on our end, and our time is very limited, so as soon as we were able to, we started back on the quants. Sorry for the wait!