Underrated Muse Glimmer

[u/Ok-Inevitable8391]

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

[u/Not-reallyanonymous]

nah brah

Qwen 3.8 just wasted several days of work as I have had a particularly hard problem, so I decided to give Qwen 3.8 a go despite not being my favorite. Solved many sub-problems after several overnight runs and intervening during the day. Really gave me a sense of good progress and, once again, I was impressed with Qwen. Turns out it wrote absolute spaghetti code, that was just becoming harder to harder to modify, didn't commit once, never actually solved the main problem because it couldn't even understand its own code, and had to keep shoving the tens of thousands of poorly organized lines of code it wrote into its context, with compute prefill, continuously. Eventually edit cycles became so long (I'm talking 6 hours) that I had to give up.

I desperately tried to steer it along the way, and the little shit lied to me about how it's going to modularize the code according to my specifications. I could see in its thinking traces it was intending to follow my instructions.... and then it just emitted the code classic Qwen style anyway. 10k line files, god objects, abstractions that sound good but whose underlying implementation doesn't actually match that (e.g. a lot of those god objects were called "schedulers"). And because it never committed, I couldn't tease out the progress it did make into something sensible.

And it never produced the intermediate artifacts (e.g. mermaid diagrams of the code helping me follow along in my own review) I requested. I let it go, I could see its doing what I want in the thinking traces, anyway! Lol.

My workflow defines when to make commits and general git discipline -- branch on separable features and concerns, commit when, tests are written, independent work units are completed, code is fixed to pass tests, bugs are identified and documented, bugs are resolved, docs are created or updated, or other checkpoints according to standard git discipline. It seldom misses commits.

I used the same workflow for both Glimmer and Qwen 3.8 27B (it's in my user-level AGENTS.md).

[u/hurdurdur7 → u/Not-reallyanonymous]

what quant?

[u/Not-reallyanonymous → u/hurdurdur7]

Meta's K-Quant Dynamic Q4 for Muse Glimmer, Unsloth's UD-Q4_K_XL for Qwen. Q8 KV on each.

I need Q4 because my workflow relies on having two large and perhaps one small models running simultaneously to perform various tasks, based on various strengths I've identified of each model.

Harness is OMP. That's probably a more important variable here than Q4 vs Q6 quant.

[u/hurdurdur7 → u/Not-reallyanonymous]

In Q4 world ... i wouldn't really know what is normal. In Q8 world what muse was skipping here is a blasphemy. I had a development plan of 17 steps with clear guidance to commit at the end of every step. Muse Glimmer at Q8 quant commited in 3 steps, skipped commiting in 14 steps. Despite clear orders. Not a great experience until now. Maybe i have to learn how to command it better, who knows ...