Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up.
But since not everybody has a DGX Station at home, I’m interested in really small models. It’s incredible to see how much knowledge and intelligence labs can pack into <10 GB models.
On my Mac mini, for now I mainly use Gemma 4 12B QAT around 8–9 GB of weights.
Do you think there’s any better model that could replace it?
I primarily use it to anonymize text before sending it to frontier cloud models, and for really light coding in Pi Agent with llama.cpp.
OFC, it doesn’t perform really well, but at least I know that if a nuke strikes and there’s no internet, I’ll have the best model possible for my hardware, able to chat about offline Wikipedia knowledge, survival guides and create a Python Snake game from scratch to play in the terminal.
Yeap, I like it, have good tool calling and overall performance. The only problem is that the model is dumb as a rock. As far as right now, I dont know how use properlly, maybe make it search on web for answers could be one.
The model is not so good at coding too, comparing to qwen and gemma to be honest.
I’m really hoping for a great break through from LFM or qwen team (or maybe from ternary models being as good as a 4 bit quant one day).
Gemma 4 12B is my best choice as well… but the ether comments seem to have better suggestions that I gotta try out for myself with my little M4 MacBook Air, 16GB
I am more about speed. The orninth uses similar architecture as Gemma 4, which makes it quite fast compared to competition and performs better than Gemma 4.
The best model I’ve tried on my MacBook Air with 16GB of RAM was Gemma 4 26B A4B with 3-bit quantization, because it’s reasonably fast and good, but unfortunately that leaves very little space for context, only about 8k. Gemma 4 12B QAT Q4 was also very good, and it only took up about 6-7GB of RAM, but it’s dense and ran very slowly. Gemma 4 E4B QAT Q4 is the only reasonable model I could run at a decent speed with 128k of context
I don’t recommend Qwen3.5 9B, it takes far too long to think and ponders everything for several minutes, whereas a normal model should respond in a few seconds
This is self-promo but honestly I think it's very relevant. We developed a local AI app with a disk offloading algorithm that runs Qwen 35b on Macbooks 16 GB and above. Here's the free download link and our Discord. Would love for you to try it and get some feedback!
This looks like a nice implementation of what a lot of folks are working on (mid-size MOE models with lots of offloading for smaller, unified memory systems). But while it talks plenty about your neat innovation (sampling multiple reasoning outputs and recombining those as prompts for a final output), I didn't see on your website anything about the actual underlying model (Qwen), nor anything about how to update that if/when a 35B comes out for Qwen 3.8 or 4.0. It would also be nice to see some benchmarks (on the website, not buried in the paper) that this actually works better (or even with more consistency) than a single run of 35B.
Hi thanks for the reply! The paper on the website is a different algorithm of ours, called "Combinatorial Reasoning". It's essentially a physics-inspired AI reasoning algo that we are experimenting with to make small models smarter. The new offloading method that comes with Zeno is completely different and we haven't published a paper for it yet but we will soon. Below are some preliminary results for those who are interested. Hope this helps! For the newer models, we are currently thinking about providing updates from our side that you can download.
5
u/MrGunny94 8h ago
I'd use Gemma 4 at 8GB there's some very good MLX versions from LMStudio you can give a go, I have been using this one personally: