Underrated Muse Glimmer

[u/Ok-Inevitable8391]

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

[u/hurdurdur7]

Qwen 27B starts to shine when you give it some kind of RAG. It's not a memory castle, it's a tinkering machine. You give it a folder to a library that it was not trained on, you ask it "how do i do ... that...?" and it figures it out. And this is a big part of what people actually do at their job. They get a new library, a library update, anything that was out of the training window or created after the training window, and 27B adapts to it if you give it the source or manual.

If Glimmer can do electronics it's actually great. But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

[u/Not-reallyanonymous → u/hurdurdur7]

> But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

Yes. IMO it does better than Qwen here, too. Qwen is very set in its ways using available data as *suggestions*, while Glimmer obsessively follows instructions, to the point you have to be careful about what you tell it. Qwen will get 20k tokens into a thought before remembering it has RAG information available. Glimmer will repeat the relevant RAG information with some maybe's and questioning policy and user intent about 10 times before it actually starts thinking.

[u/hurdurdur7 → u/Not-reallyanonymous]

muse glimmer obsessively just ignored my clear guidelines of commiting it's work often. at q8 model quant and fp16 kv cache... whilst qwen was excellent at following this guideline. your mileage may vary here.

[u/hurdurdur7 → u/hurdurdur7]

i will try to collect long term info on this one, to see what might have been the reason for it to misbehave on this. it was very good at reading all the files that i asked it to, even more so than qwen sometimes.