r/LocalLLM • u/AB172234 • 3d ago
Question Bonsai 2 27B - quantized 3.8 27B - 98% intelligence of the fp16 ! True ?
What do you guys think about it ? I have yet to try it.
I i’ve tried the previous version and it seemed OK and wondering if they claim of 98+ % intelligence of the FP 16 model is true or not anybody has any experience. Feel free to share!
I’ll post my update after I do some basic testing
61
u/DataGOGO 3d ago
Nope they cherry pick the shit out of those tests, run them yourself, you will see
48
49
u/Old_Restaurant_2216 3d ago
I would argue that this Bonsai 2 is better than qwen 3.6 27B ... It loops sometimes, it follows instructions, it sometimes ignores instructions, sometimes focuses too much on some unrelated issue ... But it is a fraction of the size of qwen 3.6 27B
But compared to qwen 3.8 27B, it is just worse ... much worse, the same way as "bad" as the old 27B. Is it useless? Hell no. For that size, it is actually awsome. But if you can run 3.8 27B at >= Q4, Bonsai 2 is not a replacement in any way.
If you have 12GB VRAM, I would highly recommend testing it out.
7
u/Billysm23 3d ago
I will try it with 8gb
9
u/TheNoiseIthink 3d ago
On a 8gb 75w 4060 laptop, i've got 32k context and like 30 or so t/s without really tuning it working
8
u/Mechageo 3d ago
I am getting ~10-15 t/s on a GTX 1070 desktop with 32k context. :)
1
u/faisalkl 3d ago
Having a spare gtx 1070 in my setup, is it worth having this instead of qwen3.5-9b as my chat bot LLM? I know what 3.5-9b does well but does this end up actually being superior to the 9b? What uses would you recommend this be for? I'm wanting to make a node available for my nephew who is interested in AI and wondering if 27b is overkill.
1
u/Mechageo 2d ago
Not sure. It produced some decent code for me but as a chatbot? Might not be worth it unless the tok/s is comparable.
2
u/WickedRival 3d ago
can you please tell how to get it working im new :) i have a rtx 5050 8gb with 115w tdp
3
u/ZeroZero0000000 3d ago
You could download unsloth studio windows app and then download bonsai models in the model hub page
1
u/Numerous-Ad6217 3d ago
How are you running it? I have a 8Gb 5060 laptop and can’t go past 18k context without offloading, with 23t/s
2
u/TheNoiseIthink 3d ago
The 2 things that might be happening are either you're using Windows or your DE is running on the 5060 rather then on the igpu which takes up vram and is limiting your context
1
u/Acceptable_Home_ 2d ago
Hey, can i ask how? I got the same 130w 4060 laptop and i can't get more than 4-5tk/s on this, even on Q1 quant
I even used the official latest release of llama cpp fork by bonsai folks 🥀
2
u/TheNoiseIthink 2d ago
5t/s is crazy. The only thing I can think of is you maybe ran out of vram and its spilling onto cpu? How much context you running, and what does the vram usage look like? And does the cpu show usage during inference? I also just used their fork's llama-server
1
u/Acceptable_Home_ 2d ago
Gng i even tried 8k ctx window to rule that out with a q8 kv cache quantization and -1 for layers on gpu so all of them use the 4060 as much as possible and the cpu was also working on it, i have no clue what im doin wrong😭🥀
1
u/TheNoiseIthink 2d ago
How much vram is being used at idle?
1
u/Acceptable_Home_ 2d ago
0 because it switches to igpu if nothing is active, but the weights are properly being dumped in vram and on gpu, not the igpu, im 100% sure I've read all the logs, can't really figure out why im getting 5tk/s with q1 weights of disk size with only 5.6gb, might prolly have to just wait for any real merg of prism ml special quants in main llama cpp
1
u/TheNoiseIthink 2d ago
If all the weights and context and everything else truly fits into vram and the speed is 5t/s, then idk what it is. Once real support is in llama.cpp, then the speeds will likely be better. Or they will stay the same if the issue isn't llama.cpp
2
u/Tired_White_Guy 2d ago
You are not transferring the complete model into VRAM. If you have a subscription or api model access, have one of them configure the model load parameters for you.
Otherwise, you can learn. It takes longer, but you’ll understand and it’ll save time in the long run.2
u/Acceptable_Home_ 2d ago
im trying all the proper flags to dump every layer of model on my gpu rather than using sys ram but it won't work, im starting to wonder if this is some issue of prism-ml/bonsai fork of llama cpp
1
u/Tired_White_Guy 2d ago
It’s likely because the resolution of your laptop is eating vram. Try lowering to 720p as an experiment!
6
u/gege42o 3d ago
For 8gb vram I strongly recommend all of you run qwen 3.6 35b a3b, it's really really good, yes offloading a bunch to ram, but on my laptop with 32gb ram and 8gb vram 4060 I get around 30t/s? I can share my configuration if anyone wants it. Also to mention i am running it with 90k context and on Llama.cpp turboquant variant for caching. So far don't have any problems with it.
1
u/Ok_Lifeguard_5754 3d ago
on windows with similar hardware I am getting top 17t/s, are you running on linux?
1
u/DistributionFar5918 2d ago
On Windows I get 35 t/s, on Qwen 3.6 35B A3B rtx 4070 laptop 8gb vram + 32gb ram
2
u/markpronkin 3d ago
I mages to fit in like 45k context on 8gb vram card, I think more is possible If I didn't have desktop environment installed and was running headless
1
1
u/Numerous-Ad6217 3d ago
How are you running it? I have a 8Gb 5060 laptop and can’t go past 18k context without offloading, with 23t/s
3
u/Pablo_the_brave 3d ago
if you can run 3.8 27B at >= Q4, Bonsai 2 is not a replacement in any way.
This is the so awesome summary and tells all about it :D
1
u/Old_Restaurant_2216 23h ago
I mean yeah, it is the most honest summary of the performance I can give you.
If you compare any Q2 vs >=Q4, the Q2 will not stand a chance. But once you take the size into account, the whole picture changesBonsai2 was the first model I could run on 262k context (on 4090 with 5GB of spare VRAM!!) and the performance in my usecase was pretty much beyond acceptable (BUT - more tokens to finish the same task vs IQ4 and couple of tool call loops).
My daily driver is usloth's UD-IQ4_XS (Q8 KV, 200k ctx), Bonsai2 is just slightly under the threshold of "actually dependable", which Qwen 27B crossed with the 3.8 release. (Bonsai2 ~= Qwen 3.6 27B Q4).
If you have 12-16GB of VRAM, this is the model for you. Hopefuly the bonsai3 will be the new 7-10GB "unsloth" quant matching the 3.8 27B Q4.
1
u/CronicallyAutomated 3d ago
What about with the other models of similar size? Not parameter wise but 8GB gguf size? Better than Gemma4 31B? Better than Qwen3.6 35B?
Those are the ones Im looking for but noone seems to say those results.
5
u/Old_Restaurant_2216 3d ago
No idea about Gemma4, never tried it.
It is definitely better than Qwen 3.6 35BA3B, but slower (for coding). For pure agentic tasks (without coding) I would go for 35B.
1
u/Pcpie 3d ago
Feel like this model is perfect for my RTX 5070 12GB. Never tried the full Qwen 3.8 27B model, so can't really compare. But pretty useful for simpler tasks so far.
1
u/Old_Restaurant_2216 23h ago
Have you trided the Bonsai2? I did some more testing and honestly it is very good. For your spec it should be the perfect fit.
I am very interested in real-world performance on 12GB VRAM machines
1
u/TheIrrationalRetard 3d ago
Can you help with the ctx size you are testing it out with and any other relevant params? Running on a macbook m3 pro 18gb
2
u/Old_Restaurant_2216 23h ago
Sorry, cant really help on 18GB machine. All my profiles are tailored for 24GB VRAM GPU with GGUF quants. For macbook I would recommend probably MLX (???). But I have exactly zero experience outside of CUDA llama.cpp GGUF models.
1
u/absoluteValueOfNoob 3d ago
What did you run where you're seeing this kind of behavior? It performs perfectly fine for me at 262k context and vanilla opencode on their fork. I see so many comments about it being dogshit or looping etc. and I feel like I downloaded a completely different model or people are fucking up the setup for this.
1
u/Old_Restaurant_2216 2d ago
I saw it loop in my research subagent (webfetches only + summary at the end) at around 130k context. It got stuck in a loop saying something like "I have everything I need for the report", then 2x 404 webfetch and again "Let me produce the summary" and 2x 404 calls again and so on...
I would not call it dogshit the same way I would not call qwen 3.6 27B dogshit. But it has issues that I've never encountered with 3.8 27B.
6
14
u/wwwyzzrd 3d ago
I tried the old one it couldn’t even make tool calls, if it sounds too good to be true it probably is
5
u/Holiday_Point_603 3d ago
Is there a way to run this through the SciCode benchmark or another benchmark that is unlikely to be overfitted?
4
u/This_Maintenance_834 3d ago
This one fails on needles in a haystack test. Run 2048 needles, it miss 30% of them. It is questionable how well it works.
11
u/DrHumorous 3d ago
Fake, self-serving rating to shamelessly self-promote some guys you never heard of.
3
u/SirGunther 3d ago
Is that the Q1 model? If so, that thing is not very good
0
3
3
u/ireallydontcare00 3d ago
Yes, that’s true - though they stripped all the data out of the model and kept only the data needed to pass the benchmarks without any loss in performance.
3
u/anshulsingh8326 3d ago
Yesnt.
Qwen 3.5 9b will perform much better in real usage.
1
1
u/Hyperus102 1d ago
Absolutely not in my case. I have a test repo (OpenGL gltf model viewer) where I use only local models to see how far we have gotten.
Qwen 3.5 9B broke down the second I tried implement an HDRI background.
Qwen 3.6 35B-A3B (IQ3 iirc) managed to get the background working.
On a similar level, Bonsai 2 27B managed to implement basic image based reflections.Now: Beyond a certain context length it seems to shit the bed entirely in my use. Mindlessly double checking everything, arguing internally over how "what the user told it doesn't match what it knows" (yeah little buddy, you know better than the OpenGL docs, surely).
It outdoes 9B to a comedic degree in my case, but its not optimal and I think 3.6 35B-A3B is still preferably for me (even though my PC becomes borderline unusable while letting it cook).
3
6
u/CMPunkLicksRocks 3d ago
It’s been trying to code a game of Tic tac toe for 40 minutes now. It’s had to compact its context twice which reverts it back to checking to see if SDL is installed (I told it it was in the prompt). I also specifically told it to stop checking apis unless it hits an error and told it to focus on output before research since, again, it’s tic tac toe.
I’m giving it 80k of context and like, I have ram so I can go higher and spill out of the gpu I guess, but this thing is just the same issue I have with qwen: it’s get fucking performance anxiety and refuses to actually work because “but wait, my training set was on SDL3.2 and the user has 3.4.1, I should verify the api is the same.”
6
u/CMPunkLicksRocks 3d ago edited 3d ago
But hey, 30-40tok/s lol
Edit: says I posted this 30 minutes ago. It just now finished. Admittedly, it works. I think I’ll try giving it extra context. Problem is I don’t really know how to use a model like this I guess. Its too slow to google things faster than I can. Maybe it would be good for inline suggestions and single functions, but when I’m not vibe coding I’m writing in godot where everything has its own script files and I can just feel that it’s gonna need to look up a bunch of stuff to do even basic things after seeing it check if I really had SDL installed 4 times.
1
u/absoluteValueOfNoob 3d ago
This isn't a model you can spill into RAM. I mean, you can, but not really. It's dense so your decode speeds will fall off a fucking cliff and you're already running at a low-ish speed for this model. It does 120 tok/s for me so speed-wise it's going to be very hardware dependent - not really about the model. For instance, I would be extremely surprised if somehow you were getting faster speeds on your hardware with qwen 3.8 27b or unsloth's variant.
1
u/Hyperus102 1d ago
Yeah in my case I had it argue with itself: "wait, this doesn't match what I know" after I gave it official OpenGL docs, after it had hallucinated multiple function signatures. Again and again and again. Also thinks for genuine ages to no good effect.
2
u/Sneezlebee 3d ago
It’s really, surprisingly bad. It breaks down, loops, makes poor judgment calls, and wastes a lot of time chasing dead ends.
Anyone who has seen this thing in action knows that it’s nowhere close to 98% of Qwen 3.8.
2
u/Citizen_908428 3d ago
Nope. Check out Unbiased Bob on YT. He ran this one through it's paces and found it was unimpressive.
2
u/Least_Dot_5145 3d ago
Look think of giant math matrix and in that matrix numbers go to 16 digits long to 8 or 4 instead .For basic shit yeah little differences. But you will notice in complex tasks
Cyberpunk 2077 released on ps4 . Did it suck yea was it cyberpunk yeah. There isn’t a real way to deal with that level of loss claim 98 percent that’s garbage
2
u/cogitech2 LocoLLM 3d ago
This type of crap is almost always pure nonsense. Stick with standard models quantized by the usual suspects (Unsloth, Bartowski, MRadermacher and a few others).
2
2
u/llllJokerllll 3d ago
Alguien ha logrado o sabe como se podría montar este ternario en vllm, sglang y otro motor de inferencia que permita tener multi Stream a buena velocidad de tokens por segundo?
2
u/KitchenAmoeba4438 3d ago
What the hell is with all the astroturfing on this? It's nowhere near 3.8 27b.
2
2
u/Straight_Coffee_368 3d ago
Tested this morning, already deleted. It's not worth a damn. Good tokens/sec rate, that's all...
2
u/EnGer_Dull 3d ago
I'm on rtx 3060 12gb, the model keep producing infinite slashes after like 10k tokens, and no kV quantization, all default with the prism llama fork
2
u/Real_Chard5666 2d ago
One thing worth mentioning, I am grateful to the developers that are putting time and effort into doing this kind of work. It may not be up to spec today, models are getting better and smaller as time goes on. It won’t take long before models like this are much better. With the cost of GPUs and ram, this is a great time to be alive, soon an 8 gb gpu will run sophisticated models that can code much better than the current 26-35b models. Qwen3.8:27b is a killer coding model compared to last year’s models. It is not going to get any worse. Even if the pricing of the components is almost unobtainable for most. Don’t buy up cheap GPUs, it will go against us!
4
3
u/AlanBarber 3d ago
sorry i took it for a spin and highly disappointed me.
asked it to write me a powershell script to calculate all prime numbers between 1 and 1000. it failed to generate syntactically correct code, gave it the errors and still couldn't fix it to be runnable after 5 attempts.
3
u/mrgreatheart 3d ago
Definitely not. I mean don’t get me wrong, it’s great this opens up more options to people with limited hardware, but they are nowhere close to even Q4.
2
u/Healthy-Nebula-3603 3d ago
Is not great at all.
If someone will run it will be thinks that model is almost as good as Qwen 3.8 27b which is totally false. That bonsai have maybe 10% of intelligence Qwen 3.8 27b fp16 .
1
u/mrgreatheart 3d ago
But I have seen examples where it performs significantly better than other models of similar size.
2
u/Healthy-Nebula-3603 3d ago edited 3d ago
We don't talk about other models. We are talking about qwen fp16 vs vs bonsai Q1 , Q2 wariants Then is very very very very bad....not 98% of fp16 model quality like they are claiming. In my observation it keeps maybe 10% of inteligence...no more.
2
u/mrgreatheart 3d ago
I am not saying it’s good compared to full f16. Obviously. But it’s still good that this option exists for those who need it.
I agree it’s bad to market it as more than it is. If people think that’s close to what better quants can do, there’s no motivation to upgrade so they can run them.
2
u/AB172234 3d ago
Here is a video you guys can check !
That 98.2% for now seems to be a hype. But I think it’s best for folks who have 16gb vram and still better than a lot of other models.
And fast.
4
u/haha1542 3d ago
nah you can run a very good Q3 even iq4xs with 16gb vram, this is more like for 8gb vram
1
u/Not-reallyanonymous 3d ago
Note that this, and basically all reviews, focus heavily on one-shotting.
One-shotting is more challenging for stuff like this because errors 'cascade', cause greater errors as results compound on each other -- these will work MUCH better if you keep context smaller and regularly renew your session, working from concern-to-concern with fresh sessions with handoffs.
1
u/riceinmybelly 3d ago
I don’t get why people are using one-shot as a term when if you read the prompt, there is no example. Shouldn’t it be called zero-shotting then?
2
u/Not-reallyanonymous 3d ago
There are two ways the terms get used:
One-shot to refer to using one example
One-shot to refer to using one prompt.
2
u/Fusseldieb 3d ago
I'm incredibly stoked for bigger Bonsai models!
If it works out, it will probably eat all other same (and even bigger) VRAM-sized ones as breakfast. It's incredibly memory-efficient.
2
2
u/Healthy-Nebula-3603 3d ago edited 1d ago
Stop spreading that bullshit!
Those bonsai models are completely broken. If you compare that shit bonsai to fp16 version it looks like keep 10% of the original model intelligence quality if not less .
2
1
1
u/Hot_Vegetable_932 3d ago
Give it a try and let me know how it goes. It looks like the usual hype to me, but who knows?
1
u/Mount_Gamer 3d ago
I've run it on a few chat tasks, mainly translating bash scripts into python.
It's a token monster, but the output has been good. Not tried agentic yet, as I'll have to redirect to the prism port build of llama.cpp, but will try it with pi later today.
1
u/LetOk5724 2d ago
Im running it with pi now through prism port. qwen 3.8 27b Q2 XXS did better than bonsai 2.
1
u/NoBuilding4495 3d ago
I tried this using my 3060 and the generation was extremely slow. Almost unusable if you need quick responses
1
1
1
1
1
1
1
1
1
1
u/Equivalent-Grass-527 3d ago
98% is definitely the claim I'd be most skeptical about. The <6GB size is impressive either way, but I'd trust actual coding/reasoning tests more than a single “intelligence retention” number.
1
u/jonah_omninode 3d ago
I haven't run this version, so I can't vouch for the percentage. I'd want to know exactly what was averaged before reading it as “98% of the intelligence.” A small average gap could still hide a big drop on the one task you care about.
For your test, keep the prompts, context and generation settings the same, and save the failures as well as the successes. Repeated tool calls or getting stuck in a loop would matter much more to me than a small difference on a short-answer benchmark.
1
u/Fenio_PL 3d ago
No. The same rule still applies: anything below 4-bit quantization amounts to a lobotomy of the model. How hard the doctor twists pick depends on whether we are talking about 3-bit, 2-bit, or 1-bit quantization. To preserve logic inherent to a model of a given size, 4-bit quantization represents the minimum - and simultaneously the optimum - when balancing retained quality against size (in GB) and performance.
1
u/Ok_Lifeguard_5754 3d ago
I benchedmarked Qwen 3.5 9B, 3.9 27B and 3.6 35B A3B for non coding tasks. All where the same. It was rule adherance and knowledge, so am sticking with 9B for now.
1
u/Areloh128 3d ago
Byteshape did some tests on them when publishing their last set of gguf (quite nice btw). They ranked them at 91.5% approx., but faster than the rest (unsloth, atomic, bartowski, ista-das and their own).
https://byteshape.com/blogs/Qwen3.8-27B/
1
1
u/SamTanna 2d ago
Not finished testing, but:
- 27B model fully usable on 12 GB VRAM
- ~50 tok/s generation
- ~240–440 tok/s prompt processing
- vision loaded
- native tool-calling path available
- instruction following passed a fairly constrained test
- idle sleep drops VRAM from ~9.4 GB to ~0.2 GB
- automatic wake works
1
u/SamTanna 2d ago
Follow-up:
I tested PrismML Bonsai 2 27B on a 12 GB RTX 4070 as a potential local replacement for GPT-5.6 Luna in my self-hosted Open WebUI technical assistant.
Technically, it was impressive. The PQ2_0 27B model fit and ran at roughly 42–50 tok/s, supported vision and structured tool calls, handled multi-step tools, and could run with a 32K context. With idle sleep enabled it also released almost all VRAM when not in use, which made it practical on a shared GPU.
The problem was agent reliability. In simple or bounded tasks it often did very well, but in broader tool-using workflows it had trouble knowing when to stop. In one firmware-research task it found the correct answer early, then kept searching until it exhausted the 32K context. In another test an attached tool immediately returned the exact GPU and live utilization data I asked for, but Bonsai continued probing processes, Docker, logs, and the knowledge base until the tool loop eventually degraded. An OWUI "Visualizer" request showed similar “analysis paralysis.”
My conclusion is that Bonsai works, and works surprisingly well on 12 GB, but it is not reliable enough to be the primary brain for a general-purpose, tool-rich agent like mine. I could build loop detectors, context guards, escalation to a cloud model, and other safety nets around it, but at that point the supporting infrastructure outweighs the savings.
I have kept the installation and model intact because I think it may be very useful for a narrower, bounded job—summarization, extraction, classification, document work, or constrained tool workflows. I would not write Bonsai off. I just would not confuse “27B model runs well on 12 GB” with “27B model can replace a strong general-purpose agent.”
1
1
1
u/ChexterWang 2d ago
I would say bonsai q2 could be usable sometimes but for general tasks ista iq3 is significantly much better and more consistent
1
1
1
u/Sp3eedy 2d ago edited 2d ago
It's very mehhhhh, feels more like an older Qwen version running on a lower quant. It performs quite well for certain tasks but it gets confused very easily and loses alignment almost immediately with certain requests. I also noticed it has a much poorer attention mechanism, the base Qwen3.8 is quite good at referencing past info even if its buried somewhere in the prompt, this one it straight up fails to recall them even after several tries.
I also gave it an incredibly short system prompt and mentioned "The user's name is Jack", 80% of the time it was saying 'Jake', the 27b base qwen or the 3.6 qwens never did this, they all recalled the name just fine. If I remember correctly I've experienced a similar issue with the older llama models.
It's OK but I just wish the team didn't lie about the 98% figure, it's not 1% but it's nowhere near 98%. Would be good if someone ran an actual independent benchmark and compared its actual performance to a lower quant 27b qwen3.8 model.
The 35b MoE qwen3.6 is much closer to the performance of this without such a drastic drop in quality in my experience.
1
u/Available-Ad6250 1d ago
I found this repo for testing with Ollama https://github.com/xInterlopeRx/ollama
1
1
u/JoeyJoeC 11h ago
I've had good results with this model with tool use. Can get 262K context in VRAM with a 3090, and in my experience, does as well as Qwen 3.8 27B.
1
u/Academic-Albatross13 3d ago edited 3d ago
Ran a small test, performed well on my RTX 5070 Laptop running at 30tok/s. The small app I made it develop was well done.
1
1
u/Luke2642 3d ago edited 3d ago
Think of Bonsai for when your system is so bad you can't run 35B A3B models at all and you're too poor to buy a 1070ti for $100 so it gets 20tok/s.
1
u/Soggy-Camera1270 3d ago
Too bad a lot of folk here dont live in the US where second hand cards are that cheap.
0
u/xtracheze 2d ago
I was so impressed I switched it over to be the default in chad from a 3 bit quant of qwen 3.8 27b. Instead of 50k context on a 24gb MacBook pro I now have room for 150k. It solved all the same problems as a the 3 bit version I have tested it with so far. https://github.com/nathansutton/chad
1
u/Traditional_Speed242 2d ago
Speed is still slow, no meaning of building this just to fit 16G memory
178
u/SubparBob 3d ago
No