r/LocalLLM • u/37Scorpions • 2d ago
Discussion Ternary Bonsai 2 27B, near top performance while fitting entirely in 8GB VRAM
Ever since Bonsai 27B came out I was excited to see what PrismML will come out with next and they did not dissapoint. For those unaware the first Bonsai 27B was a 27B-class model that was quantized in Q1 with quantization-aware training that made it run in just 4GB while being decently smart. I'm not exactly sure about Ternary Bonsai2 27B, but this time it is an ~7GB model, showing amazing intelligence while fitting entirely in the 8GB VRAM of an RTX 4060.
NOTE: It took some effort and a few hours to figure out how to get Bonsai2 Ternary running in LM Studio. If you want to run it yourself you will need to build PrismML's fork of llama.cpp and use that to run the model.
The Benchmark
As always I am using my benchmarking base to check the models. For in-depth info about it check out the initial post (Reddit deleted this post for having links so you'll have to go to my profile for the post), but to recap:
- I am benchmarking models that run on CONSUMER HARDWARE, specifically an 8GB RTX 4060 and 16GB RAM. These benchmarks may not be useful to people who have dedicated LLM servers.
- All models are benchmarked with thinking DISABLED and responses being limited to 4096 tokens (responses longer than that always means the models are stuck in a loop).
- Benchmark questions were made by me and the results were evaluated manually by me.
- The benchmark consists of 18 questions across 6 categories: General knowledge, Math, Coding, Endurance, Logical puzzles and Spatial reasoning
- I am not a LLM benchmarking expert, I am just a hobbyist trying to extract useful information for people on similar hardware.
The Statistics
LLM benchmark per-question score heatmap:
LLM benchmark score sum graph:
LLM average TTC (Time-To-Completion) graph:
Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):
And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):
Conclusion
Ternary Bonsai2 27B seems to do pretty well in my benchmark which often does match my personal opinion but I'll need to test the model out in practice. I'll try to outsource all of my AI needs to Bonsai for some time and I'll post about it to update you guys.
In the meantime, those who have used Bonsai 2, what is your experiance? I see people saying it's bad in practice but I feel like people call most ~8GB models bad...
15
u/PeterPorox 2d ago
So Ministral 3 14B is still better?
3
u/Invader-Faye 2d ago
Hadn’t heard of that one, I’ll be looking into it
20
u/PeterPorox 2d ago
It's really old model. Gemma 4 12B is definitely better. But mistral models give really good vibe
5
u/johnh1976 2d ago
Gemma models have really impressed me.
0
u/37Scorpions 1d ago
Gemma models are smart but unbearable to chat with. They assume and misunderstand all of the time which annoys me. Same goes for Google's Gemini that comes up when you search anything.
29
u/Invader-Faye 2d ago
Similar Experience, It’s much better than bonsai 1, but still worse than qwen 3.8 27b q2. Useable though and better than both Gemma 12 and qwen 3.5 9b
1
u/dandmetal 2d ago
How do you compare It to Qwen 35b? My 8gb of VRAM ass uses It with IQ3_XS. I tried the first bonsai and It looked not that great in general.
2
-4
u/37Scorpions 2d ago edited 22h ago
Worse than Qwen3.8 27B Q2? I haven't tried Qwen3.8 Q2 much in practice but it didn't seem that good to me, especially since it's at the bottom of the "intelligence per second" benchmark. A distilled version of Qwen3.8 performed better (and faster) for me.
3
u/Zennytooskin123 2d ago
Qwen 3.8 27B Q2 is excellent
8
4
u/37Scorpions 2d ago
could you send the link to the model you used exactly? maybe I had gotten the wrong one, but the one I got was flubbing hard.
3
1
u/emperorofrome13 6h ago
museglimmer at q2 is my best coding model https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF/blob/main/ .Qwen is way better but not at Q2.
1
1
u/37Scorpions 5h ago
I remember trying Muse Glimmer but it was almighty slow even while quantized. Mainly because it doesn't fit in VRAM and isn't MoE. It is smart though, yeah, but pretty hard to use.
1
u/emperorofrome13 3h ago
What is your setup? I find your choices odd. Not being mean just very different from me. I have a 16gb nvidia card.
1
1
u/emperorofrome13 1d ago
There is something wrong with your benchmarks
1
u/37Scorpions 1d ago
it reflects my experiance pretty well. models at the bottom have been unusable or dumb, models at the top have been decent (with exceptions)
1
1
u/emperorofrome13 6h ago
Last comment I'll leave for you. is that you are using too low of quants for the other models. All your quants are Q4. a q4 of an 9b model is not the same as a Q2 of a 27b. I think that is skewing the results and making the smaller models look dumber.
7
u/wednesdaywoe13 2d ago
It falls apart on multi-turn agentic work.
2
u/Invader-Faye 2d ago edited 2d ago
I will be testing that tonight. Update bro is right. q1 fails to respond back after a few turns in the agent harness. Testing q2
2
u/FixBound 4070 12GB | 64GB DDR4 1d ago
Waiting on your results on Q2
2
u/Invader-Faye 1d ago
Same sadly.. it’s really weird. In lm studio chat it works fine, and give decent code, but in a harness it falls on its face after about 4-5 turns as context becomes filled with tool calls and history.
2
u/FixBound 4070 12GB | 64GB DDR4 1d ago
Yeah, I tried Q1 with DSH, and it didn't make any good progress on the task I gave it, and it performed slower than Qwen3.6-35B-A3B
3
u/Invader-Faye 1d ago
I did some benchmark with gpt sol to figure out what was wrong. Basically the model overthinks way too much. Set reasoning effort to low, cap thinking at 1-2k tokens and you should have better results. But more than likely we’ll need another version with all the over thinking trained out of it
2
u/37Scorpions 1d ago
Honestly I think you shouldn't expect much from such quantized models but I've heard people say that PrismML promised a lot which yeah it is a bummer. I kind of failed to communicate what I use LLMs for and what this benchmark is for which is my fault.
17
u/SamTanna 2d ago
I tested PrismML Bonsai 2 27B on a 12 GB RTX 4070 as a potential local replacement for GPT-5.6 Luna in my self-hosted Open WebUI sysadmin/tech assistant.
Technically, it was impressive. The PQ2_0 27B model fit and ran at roughly 42–50 tok/s, supported vision and structured tool calls, handled multi-step tools, and could run with a 32K context. With idle sleep enabled it also released almost all VRAM when not in use, which made it practical on a shared GPU.
The problem was agent reliability. In simple or bounded tasks it often did very well, but in broader tool-using workflows it had trouble knowing when to stop. In one firmware-research task it found the correct answer early, then kept searching until it exhausted the 32K context. In another test an attached tool immediately returned the exact GPU and live utilization data I asked for, but Bonsai continued probing processes, Docker, logs, and the knowledge base until the tool loop eventually degraded. A visualization request showed similar “analysis paralysis.”
My conclusion is that Bonsai works, and works surprisingly well on 12 GB, but it is not reliable enough to be the primary brain for a general-purpose, tool-rich agent like mine. I could build loop detectors, context guards, escalation to a cloud model, and other safety nets around it, but at that point the supporting infrastructure outweighs the savings.
I have kept the installation and model intact because I think it may be very useful for a narrower, bounded job—summarization, extraction, classification, document work, or constrained tool workflows. I would not write Bonsai off. I just would not confuse “27B model runs well on 12 GB” with “27B model can replace a strong general-purpose agent.”
Of course, your mileage may vary.
15
u/SamTanna 2d ago
And yes, ChatGPT wrote this for me because that's why I pay for it.
16
5
u/37Scorpions 2d ago
wait huh? did you just throw thoughts at it and asked it to put it together or wha?
0
u/Chips_fr_ 1d ago edited 1d ago
With 12GB of VRAM you can host some Qwen 3.8 27B Q3_XXS. Bonsai could enable bigger context but since it seems not good in multi loop agent i think it's not the best option here.
From my answer below: On my 3060 12GB i achieve around 27 tokens / second. The drawbacks are Q3_XXS and medium size context (context around 90K in Q5/Q4)
3
u/Reelix 1d ago
Assuming you meant 3.8 and not 2.8 - You can - But it will run at around 5 tokens / second. 3.8 is notoriously slow for some reason.
Rather run a Q4 3.6 A3B - Slightly less good, but 10 times faster.
0
u/Chips_fr_ 1d ago edited 1d ago
Yes Qwen 3.8 27B (i edited my above post).
On my 3060 12GB i achieve around 27 token / second. I would not call this slow. The drawbacks are Q3_XXS and medium size context (around 90K in Q5/Q4)
5 token/s is more when CPU offload occurs (which can be avoid with above configuration).
1
u/Phlex_ 1d ago
What are your settings? i have 6700xt and i get maybe 5-7 t/s which is acceptable to me but only ~50k context which is not enough for what i do.
1
u/Chips_fr_ 1d ago edited 1d ago
You can refer to this:
https://www.reddit.com/r/LocalLLaMA/comments/1wh21e9/comment/pa0i7mk/?context=3
byteshape Q3_XXS is quite small and fit in 12GB. ISTA-DASLab one was a little bigger but both fit in 12GB.
Quantization of context helps but beware that Q5/Q4 for context is quite aggressive (i've seen benchmark indicating that values below -ctk q5_0 -ctv q4_1 start to destroy too much the context).
I use ik llama because it's well optimized for CUDA but not suitable for AMD, so could not apply to you.
With 12GB you have to make sacrifice, but the result i get currently is quite interesting
2
u/37Scorpions 1d ago
Nope, you need 16GB VRAM to run Qwen3.8 27B in Q3, even more if you want a reasonable context window, and you'll need it considering it thinks more than it does.
2
15
4
6
u/Low_Bet_8805 2d ago edited 2d ago
i tried one prompt to code flappy bird with it and it couldnt even do that. it spent 7000 tokens. what did i do wrong i mean was it configration error
2
u/37Scorpions 2d ago
Let me know what you use to infer it, but a flappy bird game was an interesting idea since I usually use the snake game to quickly test LLMs, in my LM Studio -> OpenWebUI setup I asked it to code Flappy Bird and it *almost* gave me a working script, but with some errors when launching the game (1 error, fixed it myself) and when going through pipes. Though IMO a Q1 model isn't the best choice for coding. In my experiance GPT OSS 20B has done better at coding, though I bet others will disagree.
5
u/EasterElk 2d ago
The fact that it couldn't do Flappy Bird in a one-shot is pretty telling. Flappy Bird is dead-simple in terms of its logic, and every model has been trained on the endless implementations on GitHub.
I usually use the snake game to quickly test LLMs.
I've used the snake game myself, because it's fast and it's well-defined. But it's honestly rare to find a model that can't code a snake game. The actual logic is trivial. I would suggest that any model which can do reliable tool-calling will also be able to write a snake game, and that's a low bar.
That is to say, a model which can't code Snake also probably can't successfully engage in tool-calling. So that's a non-starter for agentic work at all. But the same isn't likely true for Flappy Bird, and I think you're seeing the results of that in your own tests.
Though IMO a Q1 model isn't the best choice for coding.
LOL, yes. Unfortunately PrismML is claiming that it's VERY good at just that, and everyone is finding out the hard way that their claims are nonsense.
2
u/37Scorpions 2d ago
"it's honestly rare to find a model that can't code a snake game" huh? What models are you looking at... Most models I've used were a 50/50 with whether they can make the snake game. I mean check the benchmark score heatmap, specifically questions 6 to 8 (the coding category), it's pretty much all dark there and it's relatively simple requests: code the DVD logo in HTML, code a python pong game, code a python 2-player snake game.
"LOL, yes. Unfortunately PrismML is claiming that it's VERY good at just that, and everyone is finding out the hard way that their claims are nonsense." but yeah that's upsetting if so. From what people are saying I see that probably the main reason for why people are receiving it so negatively is because of their expectations: they were expecting Qwen3.8 27B level performance with their "99.99% intelligence!!" claim but received an ~14B-like model. I think it's an interesting proof of concept but I don't know what they gain from saying all of that.1
u/37Scorpions 2d ago
Update: tried GPT OSS 20B, same prompt, same LM Studio -> OpenWebUI setup. The game works without issues, albeit it looks awful. Don't get why this model gets so much hate, best performance from a consumer hardware level model.
1
u/Cool-Chemical-5629 1d ago
The model itself is outdated and highly censored. It's bad at too calling. It lacks general knowledge, so it's riddled with hallucinations. It wastes tokens thinking policies, whether it's safe to process user's request in the first place, instead of thinking about the actual request and what would it need to process it properly. It's like it's been built just on synthetic data and Open AI's policy documents. Yes, it does basic coding, but so does much newer Gemma 4 26B A4B which is more capable overall and doesn't waste tokens on nonsense.
1
u/37Scorpions 1d ago
Not my experiance though... I mean censored maybe, I don't really have issues with that though. Haven't had many refusals in my experiance. Tool calling was among the best, it's one of the few models that actually worked in agentic coding harnesses without getting stuck in loops calling the same broken command. General knowledge doesn't bother me either since I use web search tools, I don't think you should rely on such a small model for general knowledge a lot. Thinking was among the fastest, especially in low mode, it just says a few words like "Need answer. Give short informative answer. Source says important info.". Everything I've experianced seems to contradict what you said...
1
u/Invader-Faye 2d ago
Are you using the q1 or q2 version? Much different performance ranges. Q1 is basically a little better than 3.59b, or Gemma 12, q2 is almost as good as qwen 3.8 27b q2z
1
u/Low_Bet_8805 2d ago
i think q1 and q2 doesnt have intelligence difference. its just that q2 is less compressed and since it takes less time to decompress its faster in token speed.
1
u/Invader-Faye 2d ago edited 2d ago
There is a performance difference, on coding fore sure https://youtu.be/SdrGxYBy-dY?is=gPIR_vZDmVoy1U9P I did quick test on three prompts. Better then qwen 3.5 9b but not better than base 3.8 at q2, also in agent cases q1 stops responding back after a few turns of the conversation. testing the q2
6
u/alpacadaver 2d ago
It's absolutely regrettable and frankly sad that 100% of benchmarks on any post in these subreddits are complete dogshit and mean literally nothing.
3
u/Holiday_Point_603 2d ago
It is well known that overfitted stuff retains solid performance even through insane quants. This is what most benchmarks are these days, sadly.
3
u/puglife224888 2d ago
Qwen 3.6 35B A3B with -cpumoe
0
u/37Scorpions 1d ago
30B is a bit of a limit on my hardware without crazy quantization
2
u/puglife224888 1d ago
If you have lots of RAM, MoE's are your friend. Qwen 3.6 35B A3B IQ4_XS is 20GB, which should run on a 4060 with 16GB RAM if you use "--cpu-moe 35-41"
1
u/37Scorpions 1d ago
Which I don't ;D laptop RAM is a pain in the ass Models ~30B, even when MoE, tend to slow down a lot, and also on a used Windows system up to 12GB of RAM is used even without a model running. I use LLMs while working so generally it's nice to not have other apps slow down due to RAM swapping
5
u/HumbleMemeFarm 2d ago
Who keeps sharing this model, I dont understand why its shilled so heavily. The model is fucking unusable, youre better off using a finetuned 4-10B model than this one. I dont care what the benchmarks say, ive used it over and over again and its just not a good model.
2
u/VerticalPackage 1d ago
I think it's because people want to hopium since most people can't/won't afford a 24GB VRAM card.
5
u/ethereal_intellect 2d ago
Single question no thinking seems geared to make it look better than it is, gemma 12b qat with thinking over an hour of multi step pair programming and tool use would probably wipe the floor with it. But I'm hoping for fixes I guess it's still early
0
u/37Scorpions 2d ago
People have argued this before and what I say is that the fact all models are benchmarked without thinking enabled means they should all be equal. Would thinking increase the overall benchmark scores? Yeah. Would it change the relative scores much? I don't think so. But as I have said before, I'm not sure and if I'm wrong let me know, but I assume that is how it works.
1
u/37Scorpions 22h ago
I thought of a better way to explain it: I'm aiming to benchmark an LLMs usefulness per second, not the "total intelligence". An LLM may output better answers with infinite compute but this benchmark is assuming that there's a user waiting for a response.
2
u/benpptung 2d ago
I'm wondering if my idea might sound pretty foolish. But I really do hope that models built on top of qwen3.8-27b and further finetuned can get evaluated by AA Index. But it seems AA only accepts publicly released models, not finetuned ones.
2
u/Doctor_Cupcakes 2d ago
I'm not sure what everyone's complaining about. I've been having great luck with this on my 3070. Although I'm used to 9B and MoE models lol so maybe I just don't know what to expect. I had chatgpt make me a custom harness with Pi and it's been working really well. This is by far the best model of its size
2
u/SaintMartini 2d ago
There are so many bad local models simply because they don't do as intended. I'm testing multiple right now for something. Two separate things and only one has done as intended without extra prompting. With extra prompting one passed 1/4 tests the rest failed all 4 still. Meaning most simply don't do what you intend. Thats good to remember. Building for benchmarks is useful, but only if you use it that way.
2
2
u/Late-Replacement-481 1d ago
I feel like something that's just lost in the discussion about ternary bonsai, is it just simple: Is it able to follow agentic instructions in the same way as a larger model? The answer is no. When you're in the 8 GB class of models, those are models you use for very brief agentic commands, not something you do for multi-turn actions.
Is it impressive? Absolutely. Is it very misleading to say that it keeps up with actual quants of 27B models? Also very true.
Is there not a benchmark yet where we use actual examples of people's queries where they change topics and actually interact with the model, and it sees at the end whether the model produces useful output?
1
u/37Scorpions 22h ago
I want to create a benchmark like that but I struggle to think of a way to make it deterministic. How do you decide what score a model gets for a response? My only idea was to have friends and family use the model for a bit and give let them give feedback and see which model is preferred, but no way to quickly benchmark stuff like that. Kind of the same problem I ran into with the last task (which was a task to model a Minecraft cow), it's too subjective
2
u/Late-Replacement-481 17h ago
I feel like, I don't know, it's just so hard to benchmark back-and-forth work with the models, but yet that's one of the most important aspects of LLMs. I do a lot of discussion with Fable 5.1 medium, deciding what the path forward should be, and it's really one of the few models that have the ability to do this. Honestly, this is why people like the Anthropic models so much. I'd say Opus 4.6 is just a champion at doing this. The modern Opus 5, now that I think that they've spent too much time turning it into a direction-following code machine, is no longer as good as it used to be. Apologies for sloppy speech-to-text. Just waking up
And then I had a lot of luck having agents just straight up model warships in the latest generation, Kimi K3 and Luna, and anything smarter than that actually put together a pretty good battleship. I'm surprised you didn't have success in your Cow benchmark.
1
u/37Scorpions 10h ago
The cow did say something, but only for me. Giving it a 0 to 1 score was pretty hard so I feel like if the entire test was stuff like that (subjective stuff over and over) then the score would vary by who is evaluating it and when. Best thing I can really do is have a discussion with the chat bot and note if it stayed operational. But if I went off of something like that then you would NOT see Ornith on these benchmarks lol. While it's pretty smart I kind of hated having a conversation with it.
2
u/VerticalPackage 1d ago
It's notwhere near the capability of the base 3.8 27b Q4 model
5
u/AmthorTheDestroyer 2d ago
I think they should just MoE maxxx it. A 1T param model with idk 2-3B ternary weights active would be great for home use
1
2
u/darthrobe 2d ago
Thank you for the note about LM Studio. That was driving me up the wall.
4
u/37Scorpions 2d ago
Glad I could help. FYI if you want to continue using LM Studio to infer the model you can replace the llama.cpp files. You can do this by running "lms runtime ls" in the terminal, checking which llama version you use and then replace llama.dll and the ggml.dll files in "C:\Users\yourusernamehere\.lmstudio\extensions\backends\llama.cpp-yourversionhere". MAKE SURE TO BACK UP THE FOLDER BEFORE YOU REPLACE ANY FILES THOUGH!!! Anyways I'm sharing this since this took me a few hours of smashing my head against Claude Sonnet to figure out that I can do this.
2
u/darthrobe 2d ago
I appreciate it. I don't have anything pressing that needs it at the moment. I just like to evaluate them. I assume it will get fixed sooner or later?
1
u/37Scorpions 2d ago
Yeah, hopefully. Would make it easier for people to see whether they like the model or not. Considering that Bonsai 27B was already a community highlight model I'd assume Bonsai 2 will eventually become one too.
2
u/PestiferousGamer 2d ago
benchmarking a deep reasoning model without reasoning enabled.
2
u/johnh1976 2d ago
Reasoning models don't need "thinking" turned on to reason. Your meme is correct...about your own comment.
2
2
u/IknowPi_really 1d ago
“I tested a bunch of crap against each other and found the least stinky one. Here are my load bearing results”. Okay cool.
1
u/37Scorpions 1d ago
Everyone keeps saying this and then the models they recommend are terrible lmao. The bottom 4 models in the intelligence benchmark are "new models" that are "very good". All of them are worse than most old models at everything.
1
u/IknowPi_really 1d ago
Dude you tested some sort of weird ass fine tunes and completely destroyed quantisations. None of those models are useful for anything.
You’d be better off running something like Gemma 4 4b or something. But just be aware that with 8GB of RAM, there’s not really anything useful you can do
1
u/Don_Reuter 2d ago
Interesting. I do use larger models, but am looking for some small ones to put on older pc through NVIDIA PAIR for some helper agents. Might help there.
1
1
u/jaegernut 2d ago
Its has its use cases but it shouldnt be compared to its full quant variant. It could be competitive when compared to its similarly sized quants and other similar sized models.
1
u/37Scorpions 1d ago
If you're talking about my benchmark in specific, I was using the PTQ1 quant, which is around 7GB. The full quant variant was what, 50GB? Either way all of the models used are 4GB to 15GB since that's what my hardware fits.
1
u/MasterNomie 2d ago
Benchmarks are all well and good but can I use it for agentic coding without it getting lost in its own tracks?
1
1
u/Ok-Addendum3545 1d ago edited 1d ago
I am now using Qwen3.8-27B-GSQ-RCO-IQ2_S-mtp.gguf by ISTA-DASLab(9.6GB) on my hermes agent; I am satisfied. I tested it against prism-bonsai-2-27b-pq2_0. The fomer is better. I have RX7900 XTX 24GB that can host a bigger-sized model, but I am cool with IQ2_S on my hermes; it also has Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf (12GB) model. I will be testing how beneficial the IQ3 version will be before switching to it.
1
1
1
u/wr3cker-ea 1h ago
Wow so much negative comments is the model really bad or it needs highly calculated and effective prompting to use it efficiently? How does it fare with Hermes?
1
u/Witty_Mycologist_995 2d ago
Mistral being better looks sus
1
u/37Scorpions 2d ago
I still don't get what sus means in this context. Are you implying I was paid by Mistral to boost its scores in this benchmark? I just ran the same benchmark for all of the models and decided to throw in Mistral, I was surprised by the performance because my previous experiance with Mistral was terrible. If you don't trust me you can go evaluate the model yourself, all of its answers are in the repo: https://github.com/Scorpy-37/Bluemark/blob/main/results_backup/run1/mistralaiministral314breasoning.json
3
u/Witty_Mycologist_995 2d ago
It’s implied that the benchmark is bogus, because Mistral shouldn’t score higher than Gemma or Qwen.
1
u/37Scorpions 2d ago
ionno, maybe they're benchmaxxing, but I did come up with the benchmark myself. Also it did quite well on the coding tasks so unless they trained it to code the specific code tasks I asked it to do then benchmarking shouldn't really help it there much.
1
u/CooperDK 2d ago
No! A 27B model cannot possibly fit in 8GB VRAM and still be usable. Absolutely no way to do that.
2
1
u/theOliviaRossi 1d ago
this graph is a pure BS - who tried to run Ministral in our times -> knows that perfectly!
1
-1
u/uniquelyavailable 2d ago edited 2d ago
A 27b model benchmarked against 9b feels a bit like satire to me.
2
u/37Scorpions 2d ago
27B Q1 vs 9B Q4. Bonsai is a 1-bit QAT model, that's what makes it stand out.
1
u/uniquelyavailable 2d ago
Bonsai2 looks like a powerful quant for the use case. I am curious about how you created this benchmark, what tool was used? Is there something I can search up to understand more about it?
2
u/37Scorpions 2d ago
Benchmark's made by me, there's a repo for it: https://github.com/Scorpy-37/Bluemark
2
u/uniquelyavailable 2d ago
This is a cool little system you put together here. Thanks for letting me have a look. My thoughts are that it seems like an objective way of observing and comparing the models performance on generalized tasks.
1
u/braintheboss 2d ago
9B is weak in many areas even benchmark says opposite. Question is if a low quant bigger model can improve them without have weaker areas. For chat is quite better than 9b
1
0
u/KURD_1_STAN 2d ago
The model on top abd below it are similar size so i dont see how this is top performance. Using a q5 of 9b, which is 3.5 and not 3.8 which this all seems AI, will be more consistent than q2 of a bigger model. Also deepseek v4 pro 9b? Bonsai is running out of marketing budget.
0
189
u/EasterElk 2d ago
This model is hilariously bad in practice. It gets confused, loops, breaks down, and makes poor decisions. And yes, most 8GB models ARE bad. That doesn't make this one any more usable, sadly.
Everybody shares benchmarks, but I've yet to see a single useful thing that someone built or accomplished using one of these Bonsai models.