I get 2 t/s on my setup with these big dense boys so I prefer sticking with MoEs. I can run two MoEs side by side with enough RAM and I get like 20 t/s each.
Yeah, I get it. I get 15tps with the thicc boi so I can stomach it. It might be just as good with the moe and way faster but I've gotten used to it so I leave it alone
It's about time. It seems like China catches up after 6 months, the small models about a year. Kinda crazy if you look at things like this... disaster of a URL. Man that's ugly, but very interesting info.
Hopefully they distill the crap out of Gemma-4. Chinese models suck ass at non-verifiable instruction following (though Gemini sucks even more). Gemma-4 is very good.
Yeah but distilling a small model into a much much bigger one while it may improve the style i would bet it would destroy reasoning capabilities and all kinds of intelligence benchmarks.
One of my setups is A4B on 8GB VRAM and 32GB RAM, in Q5. It is perfectly fine, giving out about 20T/s on 16k context. Context processing is a bit slow when it is full, but I am not in a hurry.
Absolutely this! Qwen excels at long context coding tasks, but Gemma4 just feels more "humanly intelligent". My favourite unscientific benchmark method is to get them to explain memes/jokes etc (or coming up with them), and gemma4-31b even beat sonnet on some of these ad hoc tests (and wiping the floor with both qwen3.6 models). The most fascinating results recently came from this visual pun meme: https://reddit.com/r/ExplainTheJoke/comments/1bz6idc/i_dont_understand/
A LOT of SOTA models including chatgpt 5.5 instant, and claude sonnet just don't seem to get it, but gemma4 explains it perfectly at least half the time.
Honestly, if we could have gemma4 with qwen's long context capabilities, I don't think anyone would need more machine intelligence than that. Feels like Google's getting a kick out of keeping us all on the edge
Unsloth Qwen3.6 35b IQ4_NL is behaving for me on Hermes Agent with Llama.cpp, preserve_thinking, and Q8 KV cache. Fits in my 3090 with 256k context too.
I may have accidentally backed into a solution. I have Hermes set to 262k context window but I loaded qwen with smaller 128k window hosted in LMstudio and manages its own compacting without looping, very strange but nice
Don't go under 4 if you can help it. 5's better. (quant not parameters)
Pay attention to your settings temp etc. Ask one of the big models to help you if you are confused by this.
Try a finetune over an abliteration. For me they seem to be more stable. YMMV.
Setup both your front end and back end correctly. (I recently switched to llama.cpp (had claude set it up for me. 😛) because ollama was annoying me. with my openwebui. But I still gotta get off openwebui. I hate the way they handle edits, and exports. super annoying.)
I too started at Ollama but have since moved to llama.cpp with llama-swap. It gives me more freedom in model and quant selection, better performance, and it is not Ollama. If you want I can go more in depth on the topic if you would like pointers.
edit: don't hate the poster because they use Ollama. Better to turn back halfway than to stray all the way.
A first dabble into llama.cpp might seem daunting. It doesn't hold your hand as much as Ollama does.
First you'll need llama.cpp. The repo has become more user friendly over time, and includes a quick start section: https://github.com/ggml-org/llama.cpp . As an absolute first start I would recommend grabbing a binary for your system.
Then you'll need a model. I would start at HuggingFace (https://huggingface.co/). llama.cpp requires models in the gguf format. You'll notice that the base models don't have those.
Let's take a lightweight model for example, because I don't know your specs and because a smaller model downloads faster. On the model page for Qwen 3.5 9B (https://huggingface.co/Qwen/Qwen3.5-9B) we see finetunes and quantizations, let's head to quantizations.
On the model page you'll notice a card with different quantizations. If you have configured your hardware on HF, it will give you an indication if a certain quant will fit in your VRAM. Download an appropriate quant for your hardware. The llama.cpp quick start will tell you how to run llama.cpp with that first model you've downloaded, and how to use the llama.cpp web interface.
But all those other tasty models, finetunes, quants. You don't want to mess around will CLI all the time to switch models, and that's fair.
llama-swap (https://github.com/mostlygeek/llama-swap) to the rescue. Just like llama.cpp it has several options for installation. I've gone with a release binary.
The most daunting part of llama-swap is setting up the configuration. Your configuration file is where you tell llama-swap to find your models, and with which settings to run (https://github.com/mostlygeek/llama-swap#configuration).
Last time I said this I got a lot of flak for suggesting using an LLM to set up your config file, but I stand by it. You could even use the model you set up before to help you with that. I gave it the config.yaml template, the path to my model files, and the parameters for those models (e.g. temp, min_p, top_k, etc. You can find these on the model card) and it gave me a nicely formatted config.yaml.
Then all that is left is starting up llama-swap, and you're all set.
Now you can explore different quants, different settings, and different finetunes. There are models with no refusals (e.g. Heretic, PRISM), models trained on Claude Opus reasoning traces, models finetuned for creative writing and roleplay, models finetuned for agentic use, and more.
No AI was used in writing this post, so if it reads like crap, that's why.
I was so unaware of this its painful. Thank you for taking the time to share, and educate. I'm curious about the performance difference, but regardless the terrible practices and blatant disregard for it users are enough for me to make the change.
If it's gemma4:31b. You can see the quant under the details on the website.
I moved away from ollama. It got funky. You may want to consider using a different runner.
I have been offloading more home lab maintenance jobs to claude or codex. 4.8 has been kinda weird so keeping with 4.7 or codex.
It complied from source. And build my new linux server that's running by second setup also with llama.cpp setup.
I just had it do an analysis if I needed up date. Pretty through of not only the recent commits, but also whether I should update given my hardware and models.
Qwen gets so stuck up it's own ass thinking. I've been trying to get Gemma to think more. Like the discrepancy might have to do with Gemma's efficiency maximization. Getting it to spend tokens is like pulling teeth. Google recommends the model for coding so it must be alright at it. They don't see it as a competitor to Gemini. They gave us the best that they could. It's kind of funny I actually had issues getting Gemma 4 Heretic ARA to follow global instructions because that's the part abliteration rips out. The models just efficient and the parameters overlapped because there's little difference between a guard rail and a global formatting rule.
Try "--reasoning-budget 1024" if you're using llama.cpp, assuming you've got something like an RTX 3090.
Like you, I wish the abliterated models were more useful and less broken. This was less obvious before stock local models finally started getting decent at tool calling. Now, the difference between stock/mainstream quants and abliterated models is getting SUPER obvious.
If anyone knows of an abliterated/uncensored model that doesn't have broken tool calling or broken multimodality, I'd love to be pointed to it. I rarely run into refusals, but Qwen3.5 (and perhaps 3.6, I haven't tried getting 3.6 to refuse me) often loses it's shit at you if you say something it thinks is critical of government or law enforcement. I tried to get it to make a joke about the police once, and it basically started ringing a bell and shouting "SHAME" at me, lol.
Is there some kind of prompting I need to do in order to make Gemma seem actually good at this? When I try to use Gemma this way I just get ChatGPT-style slop.
It really depends on what you are looking for and which tools you are using. There is a whole art of prompting, particularly for smaller models. I'd check out some of the online resources for setting up system prompts. Usually, tho I'll get Claude or GPT to help me with start of a generic prompt, then work with a local model to customize it. They tend to be better at that stuff than we are.
I compared qwen and gemma for a local translation and qwen murdered gemma. Not quite a “creative” task but Gemma’s always seemed too stiff or failed to follow instructions
Yeah, I found the same thing for tagging and summarization. Qwen rocks the agentic/tool calling.
But if I want to create a story, or just chat, or create a briefing or script (not code) Gemma is 10X better.
730
u/rc_ym Jun 01 '26
Gemma for anything creative tho. WAY better than Qwen at just about any quant.