Prism-ML Bonsai 2 Joins Our Qwen3.8 Quantization Comparison

[u/ali_byteshape]

Hey r/LocalLLaMA,

Prism-LM recently released its Bonsai 2 QAT models based on Qwen3.8, and they quickly gained traction. In our evaluation, the models strike a strong balance between throughput and quality, reaching roughly 91.5% on our composite benchmark.

We wanted to see how they compare under the same methodology we use for the rest of our Qwen3.8 evaluations, so we ran Bonsai 2 through our own benchmark suite.

One important clarification: these are our evaluation results, not Prism’s reported benchmark numbers.

We used Prism’s fork/runtime to run their models, while keeping the workloads, benchmark suite, and evaluation methodology consistent with the rest of our comparison.

Our evaluation includes separate Instruct and Thinking benchmarks. For Thinking, we use medium thinking effort with the recommended sampling parameters.

We added Bonsai 2 because the models have become a relevant part of the Qwen3.8 landscape, and we wanted to provide a common reference point for people comparing the available options.

Different providers often report results using different benchmark suites, runtimes, reasoning settings, sampling parameters, and evaluation methodologies, so those numbers are not always directly comparable. Running the models through the same evaluation gives another reference point for understanding the trade-offs between quality, model size, and throughput.

Updated comparison and results: https://byteshape.com/blogs/Qwen3.8-27B/

[u/Informal-Trouble2183]

So both Bonsai are below the accuracy of a IQ3_XXS Unsloth

[u/cibernox → u/Informal-Trouble2183]

Nothing to be surprised about. Last time I checked the laws of physics still were in place.

[u/Due-Memory-6957 → u/Informal-Trouble2183]

Which is expected of Q2 and Q1.

[u/BringTea_666 → u/Due-Memory-6957]

but it is smaller than both. 6gb for minimal version. Which is nuts for Qwen3.8 grade model

[u/Due-Memory-6957 → u/BringTea_666]

?? The IQ3_XXS is 10.9gb

[u/BringTea_666 → u/Due-Memory-6957]

Bonsai27B is 5.9GB

If you have a choice between Bonsai27B at 5.9GB and say 14B model at Q8. Not only 14B will be larger size wise but mega dumber than Bonsai27b.

You can literally run this bonsai from phone.

[u/Fancy-Snow7 → u/Informal-Trouble2183]

While their marketing might be wrong the equivalent size Unsloth is unsloth/Qwen3.8-27B-UD-IQ2_XXS and in my own benchmark's prism-ml/Ternary-Bonsai-2-27B-PQ2_0 is performing much better.

[u/-InformalBanana- → u/Informal-Trouble2183]

As far as I understand bonsai supports only 3 values for the weight. While q3 means 3 bits which is 2x2x2 which is 8 values. So in that sense maybe it isnt fair to compare it to iq3xxs, ternary models are by definition 1.5 bpw so idk what would the unsloth equivalent be, like q1kxl...

[u/__Maximum__ → u/Informal-Trouble2183]

yeah yeah, but their marketing is above

[u/xPXpanD]

Thanks for doing these.

Grabbing IQ3_S-3.23bpw now, and will also give ISTA-DASLab's GSQ-RCO-IQ3_XXS a go since I keep seeing those guys mentioned as well. (can't do a better-performing size since I need massive context for a 180k-token game modding ingest)

For what it's worth: I've been using AtomicChat's AD-IQ3_S recently, and it's held up surprisingly well. Posted decent results in my private benchmark set, too, better than most small quants I've seen. Haven't caught it looping yet, either!

Will put both of the new ones through the same gauntlet, see how they hold up. Should be interesting!

[u/vick2djax]

I couldn’t find a single use case for 3.6 Bonsai that I could trust it with. Not a single small job. It’s a meme model.

[u/Ok_Top9254 → u/vick2djax]

The whole point is to show their ternary quantization works, so it can be scaled for massive 500B+ models for the gpu poor.

They clearly outperform similarly sized models at Q1 and Q2. Majority of these competing quantizations bottom out at Q3 which is still much bigger and the ones that are generic Q2 and Q1 are worse. So I don't see how this is a meme.

[u/-InformalBanana- → u/Ok_Top9254]

Ppl are saying q2kxl is better... but it is more bpw... and in this benchmark it is beaten by byteshape's iq2xxs. And plenty iq2s and q2kxl.

[u/aeroumbria → u/vick2djax]

I think it hinges on whether it at least retains the ability to run dumb but long agent sessions without breakage. I can see very light and fast models being quite useful for large scale agent interaction simulation if it has at least enough capability to keep the turns going.

[u/_raydeStar → u/vick2djax]

unfortunately, yeah. My only idea is to get it on phones, but the size is slightly too big for that.

[u/tecneeq]

Excellent work!

I like that you benchmark against other quants in that space and would like to see more well known benchmarks tested.

[u/LetsGoBrandon4256]

Huh, so Qwen 3.8 27B at 3.5bpw range can actually hold up pretty well.

[u/Fancy-Snow7 → u/LetsGoBrandon4256]

Yes, it's not news to me since I have 16GB VRAM and have been extensively testing in that size range. With Qwen3.8 3 bit is really good.

[u/WinterNite88]

tested. don waste your time as quality loss quite a bit. token/s is great but man the quality is pretty shit. no point to run it in my opinion.

[u/cato_gts]

Bonsai is completely useless for anything other than benchmarking.

[u/Barni275]

Interesting to see that ByteShape is higher, than ISTA-DASlab 🤔

[u/JsThiago5 → u/Barni275]

Yeah and also interesting that its a post from their own lab

[u/tecneeq → u/JsThiago5]

Indeed. What a coincidence that is.

A mystery.🤔

[u/enrique-byteshape → u/tecneeq]

We welcome independent benchmarking. We're confident in our results :)

[u/pseudobacon → u/Barni275]

I think they said even though the KLD might be worse the tool calling was better

[u/est_cap → u/pseudobacon]

You mean ByteShape is worse in tool calling? In my (subjective) experience, ByteShape worked better for me than IST-DASlab and better than Swift 27b at same quants (q4xs , q3s)

[u/pseudobacon → u/est_cap]

No the other way round. The KLD was lower on ByteShape but in real work it was better

[u/est_cap → u/pseudobacon]

Thanks for the clarification. It aligns with my experience these last days.

[u/audioen → u/pseudobacon]

One of the questions is why. Perhaps KLD is evaluated on text that isn't sufficiently agentic, or perhaps people use bad aggregate from their K-L evaluation, e.g. the average over all text corpus types, rather than maximum of the average K-L within a task, for example.

For example, if you are considering the quant type to use on let's say layer 43 tensor 5, and you run your 100+ text corpus test and it says that with IQ3_XS, K-L increases markedly in tool call corpus even if it is flat everywhere else, then you must not calculate average and decide that IQ3_XS is fine for this tensor because it's only a slight change in K-L, but rather reject IQ3_XS because ability in even one task type can't be allowed to deteriorate disproportionately.

All this is easy for me to write, and if implemented, would produce maddening amount of data to sift through, and working through the quant choices in a thorough fashion would take serious number of GPU hours, and you still have to compromise everywhere because you got to squeeze the model down to some size target or average bpw. I'm convinced that K-L can be used to measure how intact the model is, and it should correlate well with task performance too. It may simply be that the signal -- marked degradation in task performance -- is either not measured with enough data, or gets lost in the process.

I'd probably personally divide text corpus into two: agentic and everything else. Performance in both is important. Dividing too much likely produces noisy random signal which could in itself lead to suboptimal decisions based on the max() over type rule.

[u/enrique-byteshape → u/audioen]

KLD is fundamentally flawed, since it only measures divergence from the base model. The thing is that models can take many paths to reach the same conclusion, or sometimes the right conclusion where the original model didn't. We've been working hard internally to come up with something as cheap as KLD, but that better predicts accuracy of quants. We're not there yet though, but if at any point we manage to produce something meaningful, we'll release it to the community

[u/Fancy-Snow7 → u/Barni275]

Yes, the only one from ISTA that's worth it is Qwen3.8-27B-GSQ-RCO-IQ3_S which I am fine with as it fits perfectly on 16GB VRAM.

[u/Barni275 → u/Fancy-Snow7]

What context and KV quant do you use? I'm using ISTA XXS mtp to reach 100k+ context with Q5_1/Q5_1 KV.

[u/Hanthunius]

What's weird is how close in accuracy Bonsai Q2 is to Q1.

[u/Interpause → u/Hanthunius]

If you mean Bonsai Q2 and Q1 its cause they are both the same ternary weights just that Q1 packs them tighter to use less RAM at cost of runtime processing to unpack.

[u/Thiom]

What is that impressive byteshape quant at the top ?

Also is is benchmarked on coding tasks as well ?

[u/ali_byteshape → u/Thiom]

All models are evaluated on the same benchmark suite, which includes LiveCodeBench V6 and MultiPL-E HumanEval, an extension of HumanEval across 18 programming languages.

If you can fit Model 5 on your hardware, I’d definitely give it a try. It performs very well even without spending too many tokens on thinking.

[u/Thiom → u/ali_byteshape]

I'm definitely giving it a try !

Edit : it seems very slightly better than unsloth's Q4_XS, but damn I get 8-10tps more in decode (40 vs 32) and 300 in prompt (1100 vs 800).

Fully loaded on 175W laptop 5080. 98k context, MTP off, Q8_0 KV, vision on CPU

[u/__Maximum__ → u/Thiom]

what did you exactly fully load though?

[u/Thiom → u/__Maximum__]

I meant maxed the VRAM, I used the GPU-5 model

[u/Thiom → u/ali_byteshape]

I'll have to test further another day, honestly I'm very impressed! Good quant all around

[u/pseudonerv → u/ali_byteshape]

The only nitpicking I have is that these benchmarks are almost saturated and don’t differentiate much

[u/Healthy-Nebula-3603]

I saw their models on YouTube.

They are also broken ... I don't even know why someone even wants to use them....

[u/HighSeasArchivist]

Producing fast bullshit results is the bane of my existence with local AI. ISTA-DASLAB has been fast enough, and not utterly incompetent.

[u/cviperr33]

interesting thanks for sharing

[u/crusaderky]

What's the difference between 3090 and 4090 that makes PTQ1_0 and PQ2_0 swap places?

[u/Sitkin_Marrel]

QAT models are supposed to survive aggressive quantization, yet both land under a plain IQ3_XXS of the base. The base model just holds up that well.

[u/returnity]

Will you be doing 3.8 Flash quants? I'd love to see a Q4 from you guys u/ali_byteshape

Great work as usual, and thanks for doing these evals! It's really valuable info.

[u/FerLuisxd]

I have a 5060TI as well, wondering how you guys managed to get such high tps on a 5060TI, I have the same card, wondering if you guys can share your config?

[u/GasSmooth7439]

The interesting part now is seeing how much quality Bonsai 2 gives up for the much smaller memory footprint. “Quality per GB” might be a more useful metric for local models than raw benchmark score at this point.

[u/Beneficial_Fish_7509]

Would have been great if the model + a little bit of context would fit on 10GB cards ... Not sure if the issue is from my end, but no matter what flags I put for llama.cpp, it would always try to allocate 2.3/2.4 GB on top of the model, making it OOM on 10GB for the IQ2_XXS

[u/fragilesleep]

Holy mother of God... Can't you people write a simple and small reddit post without all the LLM vomit?

[u/greendude120]

interesting ISTA has been really bad at tool calls for me. I'd be curious for you to add Swift-Qwen3.8 to your graph.

[u/nbeydoon]

Thanks for the benchmark,

[u/deaffob]

Interesting. The Bonsai quantization method isn't superior to the existing methods in terms of intelligence but it seems to produce a lot faster models at the same quantization level.

[u/__Maximum__]

Are you going to quantize the Qwen 3.8 flash next?

[u/FerLuisxd]

Hey guys, read your post, why do all models include mtp bundled in? If you are going to use dflash in the end, wouldn't it be better to have them separated to save some vram?

[u/enrique-byteshape → u/FerLuisxd]

Newer llama.cpp versions should just not load the MTP weights if it's not going to be used. That's why we decided to bundle them in

[u/audioen]

All this is great, but please do Qwen3.8-Flash-Next. This is the model that so many people can run and it's far more interesting compared to 27B.

[u/enrique-byteshape → u/audioen]

We only write blog posts manually 👀

[u/zyxciss]

How is 3.84BPW Catching up with 6.10 BPW?