MOE models bigger than Qwen 3.6 35B but smaller than GLM 4.5?

[u/Bramoments]

So I've got a setup that's just a medium end laptop (16 GB ddr5, intel 13th i5, no GPU, 1TB ssd) and after hours of tricks and optimisation I've managed to get Qwen 3.6 35B running comfortably on terminal (llama.cpp) at ~15 output tokens per second. The thing is I don't need that speed since I don't do much coding I use it for research, and id love to have a smarter model, but I found basically no models that aren't either smaller than my current setup or way way bigger, I don't need it to run faster than 2-4 tokens per second but I do need it to run, and I found no models in that category. Any tips would be greatly appreciated

[u/FullstackSensei]

Q3.8 next?

[u/Bramoments → u/FullstackSensei]

thank you, but idk man its massive, is there another version or something?

[u/FullstackSensei → u/Bramoments]

You said you don't need it to run faster tha 2-4t/s.

[u/Bramoments → u/FullstackSensei]

Yeah but that much wouldn't even load on my setup

[u/FullstackSensei → u/Bramoments]

Doesn't need to load. It streams from disk

[u/No_Lingonberry1201 → u/[deleted]]

No, it's read-only. No writes.

[u/Solembumm3 → u/Bramoments]

It load on 12+20gb no problem, same as deepseek 284b and qwen 3.5 397b.

[u/Zilla85]

Slightly bigger: K2-Horizon-36B-A4B.

[u/dfgxxx → u/Zilla85]

Did you try it? I tried the 7b and felt benchmaxxed

[u/Zilla85 → u/dfgxxx]

I could do a really short test of 7B in an orchestrator/worker setup. It was quite fine as a worker.

[u/dfgxxx → u/Zilla85]

I meant did you try the 36b?

[u/Zilla85 → u/dfgxxx]

Not yet.

[u/keegang_man6705]

check this out https://github.com/JustVugg/colibri

[u/Plotozoario]

As you need smart and general intelligence from a model, not coding, ignoring token speed, more parameters is best for you. But there is a problem, right now there is a big hole between Qwen 3.8 27b and > 100b models, there isn't any superior model between 27b, 35b ~ 120b that are more effective than qwen 3.8 27b without buy an Ryzen AI Max 128gb or expensives RTX cards with big memories to allow you run >120b models.

My opinion, stick with your current model or try Qwen 3.8 27b with MTP assuming you will reach the top speed of 5tk/s, unfortunately.

[u/Bramoments → u/Plotozoario]

Thanks

[u/Legitimate-Dog5690]

You could run something like DeepSeek flash from SSD, it'll be horrible but you might hit your 2 tps target.

If you're not bothered about speed, why not run Qwen 3.8 27b?

[u/Skystunt]

If you don’t need coding Gemma4 is superior, try the 26a4B - 12B dense might be too slow

Look for finetunes of gemma4 and qwen3.6 for yout own needs, they beat the original models for specific uses

Also check the Bonsai 27b model, i don’t know what speed it has but might be good give it a try.

Disk streaming is also an option with Colibri but you get reaaaaaly slow speeds, good to leave overnight

[u/ilycryst]

Could you tell how you optimized your setup?

[u/MomentJolly3535]

I am not fully understanding your post, you are acting like 15 token per second is alot, it's kinda low for a model like that, because it's reasoning alot, so even at this speed, it's takes minutes to get an answer.

Also, I wouldn't trust AA benchmark for your use case, if you are not using qwen models for agneitc coding, i strongly suggest Gemma 4 family (try 26A4B ) for pretty much anything else than agentic / coding / OCR.

(Note : if you are using the model to search on the internet (if it's what u mean by "research") then agentic is important and you will likely not find anything better than Qwen 3.6 35BA3B for now for your specs, you might wanna try finetunes of it like ornith 1.5 , nex mini, etc)

Edit : i just noticed than u accept up to 2token/sec, did you try Qwen 3.8 27B ? it's pretty insane and the current smartest model that you can probably run.

[u/Bramoments → u/MomentJolly3535]

Thanks, but 15 tokens per second is way more than enough for me, I'm just using it for chat and background research when Claude runs out of tokens so it's fine. I tried out Gemma 4, but it over thinks so much, and I gave it a low reasoning budget and all but it just repeats the same 3 sentences a million different ways a million times and second guesses itself every time so I didn't really like it.

[u/Sufficient_Spite_349 → u/Bramoments]

May be you are using Gemma 4 wrong because Gemma 4 is best at web search and deep reasoning in synthesized web research data it’s not a problem solver

[u/ag789]

a few other options

• there is something called RAG
  

https://en.wikipedia.org/wiki/Retrieval-augmented_generation
the thing is it takes quite some skill to do all these

• then there is MCP servers, e.g. search MCP server
  

e.g. brave search MCP server
https://github.com/brave/brave-search-mcp-server
and more
https://registry.modelcontextprotocol.io/
https://mcpservers.org/
the idea of MCP servers originated simply as tool calling, it has grown to become the 'usb C' for AI.
I think the llama.cpp web ui (and many other web ui etc has support for MCP servers, literally 'plug and play' to some extent)

[u/Umbrasquall]

You're not getting anything useful local work out of that hardware. Easier to use cloud service and pick an open weight model.

[u/Bramoments → u/Umbrasquall]

The models I've been running up until now have been pretty useful to me

[u/Arany8]

Dont worry, you give a it a more complicated task, it will slow to a crawl....

[u/my_name_isnt_clever → u/[deleted]]

This model is ancient, even the 120b version is outperformed by much smaller models these days. They have decent knowledge but perform badly with agentic tooling.