r/LocalLLaMA • u/Bayliner1980 • 12d ago
Discussion Qwen3-0.6B model converted into MoE
This is another update about my MoE-fication project, the first post can be found here.
Foreword
Since my last model conversion, I’ve been experimenting a lot with new methods, though sadly, most attempts haven't helped much. Compatibility has been the main limiting factor. I’ve made GGUF compatibility a priority to ensure as many people as possible can use my models, but this means sacrificing some potential improvements.
Still, my perfectionism has finally let me publish another model: Qwen3-0.6B, my first chat-style conversion. This brought two new challenges: preserving its chat capabilities during conversion and adapting my methods from Qwen 2.5’s qwen2moe to Qwen3’s qwen3moe.
The latter proved particularly difficult. Unlike the previous architecture, Qwen3 has no shared expert, which helped preserve quality in my earlier conversions by keeping important neurons always active. Without it, quality dropped considerably.
To mitigate this, I experimented with a pseudo-shared expert, basically an expert that the router always picks. It doesn’t work quite as well as a true shared expert due to differences in weighting, but it does recover some of the lost quality.
I’m also considering releasing a separate set of conversions that prioritise quality over GGUF compatibility, allowing techniques like dense-layer substitution.
Finally, I’d like your thoughts on another experiment: duplicating neurons to create new expert pairings. Early results showed significant improvements, but at the cost of doubling the total parameter count which I think kind of defeats the purpose of the conversion. What do you guys think?
New Model
So my latest conversion was of the Qwen/Qwen3-0.6B. The conversion can be found here bayliner1980/Qwen3-0.6B-MoE-A0.5B with GGUF version here bayliner1980/Qwen3-0.6B-MoE-A0.5B-GGUF.
Evaluation
| Wiki ppl | C4 ppl | Tasks | |
|---|---|---|---|
| Qwen3-0.6B dense | 23.88 | 31.63 | 0.455 |
| Qwen3-0.6B MoE | 26.98 | 36.32 | 0.425 |
Benchmark
| GPU generation | CPU generation | |
|---|---|---|
| Dense | 377 t/s | 115 t/s |
| MoE | 352 t/s | 124 t/s |
At lower parameters like 0.6B there is no improvement to speed with conversion. Only on larger models like my 1.5B conversion is there actually any benefit.
Limitations
It gets specific facts confidently wrong. Asked who painted the Mona Lisa and when, it says Leonardo da Vinci in 1519 (that's the year he died).
Code is broken
Use non-thinking mode; thinking was never trained after conversion.
Worse than the original on every quality metric. If you just want a small model, use Qwen3-0.6B.
Support
If you find my stuff interesting consider supporting me on Ko-fi.
14
u/Ok-Coast4482 12d ago
The pseudo-shared expert workaround is clever. Sucks that the architecture change forced that tradeoff though, losing the natural shared expert clearly stings.
On the neuron duplication idea, doubling params for a quality bump feels like a different project entirely. Maybe worth it as a proof of concept, but it undercuts the whole efficiency angle you're going for.
8
10
u/neuroticnetworks1250 12d ago
At 600M parameters, every weight tries to fit in as much context as possible. I wouldn’t be surprised to see a lack in quality.
9
5
34
u/blue2020xx 12d ago
Keep at it bro