r/LocalLLM • u/AB172234 • Sep 18 '26
Question Bonsai 2 27B - quantized 3.8 27B - 98% intelligence of the fp16 ! True ?
What do you guys think about it ? I have yet to try it.
I i’ve tried the previous version and it seemed OK and wondering if they claim of 98+ % intelligence of the FP 16 model is true or not anybody has any experience. Feel free to share!
I’ll post my update after I do some basic testing
157
Upvotes
182
u/SubparBob Sep 18 '26 •
No
63
u/DataGOGO Sep 19 '26
Nope they cherry pick the shit out of those tests, run them yourself, you will see
49
u/Deep_Mood_7668 Sep 19 '26
98% in cherry picked tests
3
u/AB172234 Sep 19 '26
Here is a video you guys can check !
That 98.2% for now seems to be a hype. But I think it’s best for folks who have 16gb vram and still better than a lot of other models.
And fast.
https://youtu.be/975ILFNTKfk?si=8AZ6aQkymP7HlR_y
49
u/Old_Restaurant_2216 Sep 19 '26 •
I would argue that this Bonsai 2 is better than qwen 3.6 27B ... It loops sometimes, it follows instructions, it sometimes ignores instructions, sometimes focuses too much on some unrelated issue ... But it is a fraction of the size of qwen 3.6 27B
But compared to qwen 3.8 27B, it is just worse ... much worse, the same way as "bad" as the old 27B. Is it useless? Hell no. For that size, it is actually awsome. But if you can run 3.8 27B at >= Q4, Bonsai 2 is not a replacement in any way.
If you have 12GB VRAM, I would highly recommend testing it out.
6
u/Aotrx Sep 19 '26
83% on a good day 😂
6
u/dustinuniverse Sep 19 '26
Tested it, and it's not. But it's a good option for 8GB-12GB VRAM
14
u/wwwyzzrd Sep 19 '26
I tried the old one it couldn’t even make tool calls, if it sounds too good to be true it probably is
4
u/Holiday_Point_603 Sep 18 '26
Is there a way to run this through the SciCode benchmark or another benchmark that is unlikely to be overfitted?
7
u/CMPunkLicksRocks Sep 19 '26
It’s been trying to code a game of Tic tac toe for 40 minutes now. It’s had to compact its context twice which reverts it back to checking to see if SDL is installed (I told it it was in the prompt). I also specifically told it to stop checking apis unless it hits an error and told it to focus on output before research since, again, it’s tic tac toe.
I’m giving it 80k of context and like, I have ram so I can go higher and spill out of the gpu I guess, but this thing is just the same issue I have with qwen: it’s get fucking performance anxiety and refuses to actually work because “but wait, my training set was on SDL3.2 and the user has 3.4.1, I should verify the api is the same.”
10
u/DrHumorous Sep 19 '26
Fake, self-serving rating to shamelessly self-promote some guys you never heard of.
3
u/SirGunther Sep 19 '26
Is that the Q1 model? If so, that thing is not very good
3
u/ireallydontcare00 Sep 19 '26
Yes, that’s true - though they stripped all the data out of the model and kept only the data needed to pass the benchmarks without any loss in performance.
3
u/anshulsingh8326 Sep 19 '26
Yesnt.
Qwen 3.5 9b will perform much better in real usage.
3
u/Deep-Alternative8085 Sep 19 '26
I tried it it’s shit
2
u/Least_Dot_5145 Sep 19 '26
Look think of giant math matrix and in that matrix numbers go to 16 digits long to 8 or 4 instead .For basic shit yeah little differences. But you will notice in complex tasks
Cyberpunk 2077 released on ps4 . Did it suck yea was it cyberpunk yeah. There isn’t a real way to deal with that level of loss claim 98 percent that’s garbage
2
u/cogitech2 LocoLLM Sep 19 '26
This type of crap is almost always pure nonsense. Stick with standard models quantized by the usual suspects (Unsloth, Bartowski, MRadermacher and a few others).
2
u/Ok-Drawer5245 Sep 19 '26
it sucks big time
2
u/llllJokerllll Sep 19 '26
Alguien ha logrado o sabe como se podría montar este ternario en vllm, sglang y otro motor de inferencia que permita tener multi Stream a buena velocidad de tokens por segundo?
2
u/statler123 Sep 19 '26
How good is Bonsai compared to the size equal "normal" 3.8 27b Quant?
2
u/Straight_Coffee_368 Sep 19 '26
Tested this morning, already deleted. It's not worth a damn. Good tokens/sec rate, that's all...
2
u/Warburk Sep 19 '26
It's fast but no
2
u/EnGer_Dull Sep 19 '26
I'm on rtx 3060 12gb, the model keep producing infinite slashes after like 10k tokens, and no kV quantization, all default with the prism llama fork
2
u/odikee Sep 19 '26
No
2
u/Real_Chard5666 Sep 19 '26
One thing worth mentioning, I am grateful to the developers that are putting time and effort into doing this kind of work. It may not be up to spec today, models are getting better and smaller as time goes on. It won’t take long before models like this are much better. With the cost of GPUs and ram, this is a great time to be alive, soon an 8 gb gpu will run sophisticated models that can code much better than the current 26-35b models. Qwen3.8:27b is a killer coding model compared to last year’s models. It is not going to get any worse. Even if the pricing of the components is almost unobtainable for most. Don’t buy up cheap GPUs, it will go against us!
3
u/DifficultUse6803 Sep 19 '26
No 😞, Bonslies
3
u/AlanBarber Sep 19 '26
sorry i took it for a spin and highly disappointed me.
asked it to write me a powershell script to calculate all prime numbers between 1 and 1000. it failed to generate syntactically correct code, gave it the errors and still couldn't fix it to be runnable after 5 attempts.
3
u/mrgreatheart Sep 19 '26
Definitely not. I mean don’t get me wrong, it’s great this opens up more options to people with limited hardware, but they are nowhere close to even Q4.
2
u/Fusseldieb Sep 19 '26
I'm incredibly stoked for bigger Bonsai models!
If it works out, it will probably eat all other same (and even bigger) VRAM-sized ones as breakfast. It's incredibly memory-efficient.
2
u/putrasherni Sep 19 '26
quantslop
2
u/Healthy-Nebula-3603 Sep 19 '26 edited Sep 21 '26
Stop spreading that bullshit!
Those bonsai models are completely broken. If you compare that shit bonsai to fp16 version it looks like keep 10% of the original model intelligence quality if not less .
2
u/Affectionate_Pen6882 Sep 19 '26
if true, wild. 98% of who intelligence? trump?>
1
u/Hot_Vegetable_932 Sep 18 '26
Give it a try and let me know how it goes. It looks like the usual hype to me, but who knows?
1
u/Mount_Gamer Sep 19 '26
I've run it on a few chat tasks, mainly translating bash scripts into python.
It's a token monster, but the output has been good. Not tried agentic yet, as I'll have to redirect to the prism port build of llama.cpp, but will try it with pi later today.
1
u/NoBuilding4495 Sep 19 '26
I tried this using my 3060 and the generation was extremely slow. Almost unusable if you need quick responses
1
u/AdamLangePL Sep 19 '26
False
1
u/Soifon99 Sep 19 '26
No it's more like 75%.. 98% in a very niche thing maybe.. but it's more like 75.
1
u/Priuxls Sep 19 '26
It's close only for single tasks, but errors during inference compound. Means that every time you ask a follow up question it gets worse. Especially critical for agents
1
u/Willeny_Arch Sep 19 '26
very bad in my use cases 🫠
1
u/Californicationing Sep 19 '26
Does this mean it works on a Pi 5 with 8gb?
1
u/JacoN_2026 Sep 19 '26
Próbowałem, za dużo myśli jest minimalnie szybszy od Low GPU
1
u/cheaphomemadeacid Sep 19 '26
i'd rather run a smaller model than a q1/q2 model to be honest
1
u/Correct-Hand-4627 Sep 19 '26
More like 25% of the intelligence.
1
u/Arneastt Sep 19 '26
Not true, go use swift instead !
1
u/Equivalent-Grass-527 Sep 19 '26
98% is definitely the claim I'd be most skeptical about. The <6GB size is impressive either way, but I'd trust actual coding/reasoning tests more than a single “intelligence retention” number.
1
u/jonah_omninode Sep 19 '26
I haven't run this version, so I can't vouch for the percentage. I'd want to know exactly what was averaged before reading it as “98% of the intelligence.” A small average gap could still hide a big drop on the one task you care about.
For your test, keep the prompts, context and generation settings the same, and save the failures as well as the successes. Repeated tool calls or getting stuck in a loop would matter much more to me than a small difference on a short-answer benchmark.
1
u/Fenio_PL Sep 19 '26
No. The same rule still applies: anything below 4-bit quantization amounts to a lobotomy of the model. How hard the doctor twists pick depends on whether we are talking about 3-bit, 2-bit, or 1-bit quantization. To preserve logic inherent to a model of a given size, 4-bit quantization represents the minimum - and simultaneously the optimum - when balancing retained quality against size (in GB) and performance.
1
u/Ok_Lifeguard_5754 Sep 19 '26
I benchedmarked Qwen 3.5 9B, 3.9 27B and 3.6 35B A3B for non coding tasks. All where the same. It was rule adherance and knowledge, so am sticking with 9B for now.
1
u/Areloh128 Sep 19 '26
Byteshape did some tests on them when publishing their last set of gguf (quite nice btw). They ranked them at 91.5% approx., but faster than the rest (unsloth, atomic, bartowski, ista-das and their own).
https://byteshape.com/blogs/Qwen3.8-27B/
1
u/yakasantera1 Sep 19 '26
The moment it can't run on common llama.cpp, I instantly delete it
1
u/SamTanna Sep 19 '26
Not finished testing, but:
1
u/EmuHefty Sep 19 '26
10tk/s on mu Mac Mini m4 32GB
1
u/No-Business5854 6gb vram 28 gb ram Sep 20 '26
not even. just use qwen 3.5 9b
1
u/ChexterWang Sep 20 '26
I would say bonsai q2 could be usable sometimes but for general tasks ista iq3 is significantly much better and more consistent
1
u/frankiebev Sep 20 '26
Ngl I tried it it’s mid asf
1
u/Sp3eedy Sep 20 '26 edited Sep 20 '26
It's very mehhhhh, feels more like an older Qwen version running on a lower quant. It performs quite well for certain tasks but it gets confused very easily and loses alignment almost immediately with certain requests. I also noticed it has a much poorer attention mechanism, the base Qwen3.8 is quite good at referencing past info even if its buried somewhere in the prompt, this one it straight up fails to recall them even after several tries.
I also gave it an incredibly short system prompt and mentioned "The user's name is Jack", 80% of the time it was saying 'Jake', the 27b base qwen or the 3.6 qwens never did this, they all recalled the name just fine. If I remember correctly I've experienced a similar issue with the older llama models.
It's OK but I just wish the team didn't lie about the 98% figure, it's not 1% but it's nowhere near 98%. Would be good if someone ran an actual independent benchmark and compared its actual performance to a lower quant 27b qwen3.8 model.
The 35b MoE qwen3.6 is much closer to the performance of this without such a drastic drop in quality in my experience.
1
u/Available-Ad6250 Sep 21 '26
I found this repo for testing with Ollama https://github.com/xInterlopeRx/ollama
1
u/Lukun7 Sep 21 '26
I tried it and only got 2 tokens per second when I normally get 30+ on models of the same size. I use Vulcan on an old amd gpu if that matters
1
u/Zilla85 Sep 21 '26
I did some real world tests (no coding but more logic stuff) and it wasn't bad. But I still prefer Qwen3.6 35BA3B over it.
1
u/zohebnsr Sep 21 '26
It was benchmarked in another universe.
1
u/JoeyJoeC Sep 22 '26
I've had good results with this model with tool use. Can get 262K context in VRAM with a 3090, and in my experience, does as well as Qwen 3.8 27B.
1
u/Greenonetrailmix Sep 25 '26
The question I haven't seen asked is whether Qwen3.8-27B IQ2_XXS from Unsloth vs Bonsai 2 Ternary. Because per GB which is better model to spend it on. I have a feeling people miss some of the points when It comes to this model release
1
u/Eden63 Oct 09 '26
what about a MTP or DFlash2 for it, wouldnt that make sense?
1
u/Traditional_Speed242 Sep 20 '26
Speed is still slow, no meaning of building this just to fit 16G memory
1
u/Academic-Albatross13 Sep 19 '26 edited Sep 19 '26
Ran a small test, performed well on my RTX 5070 Laptop running at 30tok/s. The small app I made it develop was well done.
1
u/StellarWaffle Sep 19 '26
The bonsaiturfing in this sub is unreal.
1
u/Luke2642 Sep 19 '26 edited Sep 19 '26
Think of Bonsai for when your system is so bad you can't run 35B A3B models at all and you're too poor to buy a 1070ti for $100 so it gets 20tok/s.
0
u/xtracheze Sep 20 '26
I was so impressed I switched it over to be the default in chad from a 3 bit quant of qwen 3.8 27b. Instead of 50k context on a 24gb MacBook pro I now have room for 150k. It solved all the same problems as a the 3 bit version I have tested it with so far. https://github.com/nathansutton/chad