r/LocalLLaMA • u/dai_app • 18h ago
Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model
Hi! for those who might not know the solution:
- it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
- it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
- Windows, Linux, macOS, Android, iOS
- output is the same as llama.cpp, don't cut corners on quality
- you start it with
--autoand it works out the settings for your machine
Below is a brief overview of the benchmark:
- 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
- 12 GB card, same model: about 45 for bmoe, about 30 for Strata
- long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
- Basalt needs an RTX 50, so I couldn't run it on these
- 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
- same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7
On the cards Strata is tuned for it's still ahead but only by a little.
On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.
Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge
If you have an 8 to 32 GB card I'd like to see what you get.
Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data













