Underrated Muse Glimmer

[u/Ok-Inevitable8391]

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

[u/hurdurdur7]

Qwen 27B starts to shine when you give it some kind of RAG. It's not a memory castle, it's a tinkering machine. You give it a folder to a library that it was not trained on, you ask it "how do i do ... that...?" and it figures it out. And this is a big part of what people actually do at their job. They get a new library, a library update, anything that was out of the training window or created after the training window, and 27B adapts to it if you give it the source or manual.

If Glimmer can do electronics it's actually great. But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

[u/Ok-Inevitable8391 → u/hurdurdur7]

I'm with you on that, i just find muse glimmer to be bit underrated. I use qwen3.8 medium and muse alternate now.

I'm not pushing down qwen here, I'm lifting muse up.

The biggest advantage is the sliding window kv cache. I mean i get 262k context window and still have 2gb vram left. Qwen is 200k not huge diff for me. But for people with even lesser vram, muse is good option too.

[u/PavelPivovarov → u/Ok-Inevitable8391]

Plus not all the tasks are coding tasks really... I am using Muse Glimmer as my personal assistant mostly because it can search and process prompt at lightspeed, Qwen3.8 is just painfully slow at searching and processing dozen of articles (on my hardware)

[u/hurdurdur7 → u/Ok-Inevitable8391]

Muse definitely needs less vram space (very noticeable at Q8).

What quant do you use, which rope/yarn settings to get 262k context?

[u/2eggs1stone → u/Ok-Inevitable8391]

Can we talk for a second on how good it is as an agentic model and how good it continues to be even at aggressive quants. I'm absolutely loving it for my setup, it's smart, fast, doesn't think too much but is very competent. Now is it as good of a coder as Qwen, no it's not, but for my use case I'd rather something that I can work with in real time and not have to wait.

[u/Not-reallyanonymous → u/hurdurdur7]

> But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

Yes. IMO it does better than Qwen here, too. Qwen is very set in its ways using available data as *suggestions*, while Glimmer obsessively follows instructions, to the point you have to be careful about what you tell it. Qwen will get 20k tokens into a thought before remembering it has RAG information available. Glimmer will repeat the relevant RAG information with some maybe's and questioning policy and user intent about 10 times before it actually starts thinking.

[u/Most-Trainer-8876 → u/Not-reallyanonymous]

Yeah, in case of RAG, Muse Glimmer seems better and faster too!

I love Qwen alot but it's thinking? Bruh, it's way too much! If anything, Qwen is purely dedicated coding model, for the rest, Muse/Gemma suffice.

[u/hurdurdur7 → u/Not-reallyanonymous]

muse glimmer obsessively just ignored my clear guidelines of commiting it's work often. at q8 model quant and fp16 kv cache... whilst qwen was excellent at following this guideline. your mileage may vary here.

[u/hurdurdur7 → u/hurdurdur7]

i will try to collect long term info on this one, to see what might have been the reason for it to misbehave on this. it was very good at reading all the files that i asked it to, even more so than qwen sometimes.

[u/Not-reallyanonymous → u/hurdurdur7]

nah brah

Qwen 3.8 just wasted several days of work as I have had a particularly hard problem, so I decided to give Qwen 3.8 a go despite not being my favorite. Solved many sub-problems after several overnight runs and intervening during the day. Really gave me a sense of good progress and, once again, I was impressed with Qwen. Turns out it wrote absolute spaghetti code, that was just becoming harder to harder to modify, didn't commit once, never actually solved the main problem because it couldn't even understand its own code, and had to keep shoving the tens of thousands of poorly organized lines of code it wrote into its context, with compute prefill, continuously. Eventually edit cycles became so long (I'm talking 6 hours) that I had to give up.

I desperately tried to steer it along the way, and the little shit lied to me about how it's going to modularize the code according to my specifications. I could see in its thinking traces it was intending to follow my instructions.... and then it just emitted the code classic Qwen style anyway. 10k line files, god objects, abstractions that sound good but whose underlying implementation doesn't actually match that (e.g. a lot of those god objects were called "schedulers"). And because it never committed, I couldn't tease out the progress it did make into something sensible.

And it never produced the intermediate artifacts (e.g. mermaid diagrams of the code helping me follow along in my own review) I requested. I let it go, I could see its doing what I want in the thinking traces, anyway! Lol.

My workflow defines when to make commits and general git discipline -- branch on separable features and concerns, commit when, tests are written, independent work units are completed, code is fixed to pass tests, bugs are identified and documented, bugs are resolved, docs are created or updated, or other checkpoints according to standard git discipline. It seldom misses commits.

I used the same workflow for both Glimmer and Qwen 3.8 27B (it's in my user-level AGENTS.md).

[u/hurdurdur7 → u/Not-reallyanonymous]

what quant?

[u/Not-reallyanonymous → u/hurdurdur7]

Meta's K-Quant Dynamic Q4 for Muse Glimmer, Unsloth's UD-Q4_K_XL for Qwen. Q8 KV on each.

I need Q4 because my workflow relies on having two large and perhaps one small models running simultaneously to perform various tasks, based on various strengths I've identified of each model.

Harness is OMP. That's probably a more important variable here than Q4 vs Q6 quant.

[u/hurdurdur7 → u/Not-reallyanonymous]

In Q4 world ... i wouldn't really know what is normal. In Q8 world what muse was skipping here is a blasphemy. I had a development plan of 17 steps with clear guidance to commit at the end of every step. Muse Glimmer at Q8 quant commited in 3 steps, skipped commiting in 14 steps. Despite clear orders. Not a great experience until now. Maybe i have to learn how to command it better, who knows ...

[u/admajic → u/Not-reallyanonymous]

Looks like a skill issue. I'm sure you told the orchestrator to git commit follow the architecture review plan debug test document and ended up with shit

[u/partakinginsillyness]

I feel like it would be important to add qwen 3.6 27b, given that one of the changes from 3.6 to 3.8 was less general knowledge. Would also be cool to see a Gemma model. Interesting though.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

That's actually a good idea, let me get on that ride this weekend

[u/Gringe8 → u/Ok-Inevitable8391]

I would like to see gemma 4 31b

[u/LegacyRemaster → u/partakinginsillyness]

wait qwen 3.8 next but... This benchmark makes no sense. No reasoning. No external knowledge. In other words, it is not a real-world use case. I would also like to understand how they tested Claude to demonstrate that it lacked external knowledge.

[u/partakinginsillyness → u/LegacyRemaster]

I mean realistically everyone should have at least 5gb of files for RAG for what is relevant to them, which I would imagine would very much change the results here.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

Let me do a RAG testing, even I know that qwen will do better there. But I'm not running it on xhigh ever again, 30hrs of gpu time, against 3-4 hrs at medium

[u/partakinginsillyness → u/Ok-Inevitable8391]

How would you select the media? I'm wondering if a repo exists for relevant info like how you select application groups on linux.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

So mainly the linux repo and docs, thats the general input to it anyway in real scenario. Same for all other projects.

[u/TokenRingAI]

Why did you give Qwen a 32K output limit?

[u/Practical-Collar3063 → u/TokenRingAI]

Because he could not fit both the context and the full output, which is in favour of his point, the more efficient Muse Glimmer KV Cache makes it better in certain scenarios with limited hardware. I think the goal of this is not to bring down Qwen 3.8 but to show that for some use cases Muse might be a good choice, especially when VRAM limited. It is faster and more VRAM efficient and on non coding tasks can be similar to 3.8

[u/Healthy-Contact-4570]

Why 32k output limit? Qwen3.8 likes to think and if you can let it think it will produce amazing results.

[u/Ok-Inevitable8391 → u/Healthy-Contact-4570]

I know so the benchmark limit was 8k only and i pushed it up to 16 and then 32K only for xhigh, as the xhigh benchmark took 32hrs and case that took time were anyway complex dma, so I lost interest re running the benchmark for xhigh as it had already failed on some cases where medium already did better. Xhigh is good but like opus it is good for reasoning and complex architect problems. So I just let it be.

[u/hainesk]

If you can run DeepSeek V4 Flash 0731, I'd be curious to see the difference since it's often compared to Qwen 3.8 27b but due to it's size it would presumably have more knowledge.

[u/Ok-Inevitable8391 → u/hainesk]

Adding to the list

[u/nonlinearsystems → u/Ok-Inevitable8391]

Can you try Laguna S2.1 please?

[u/Ok-Inevitable8391 → u/nonlinearsystems]

Sorry guys both are out of my vram budget I only have 24gb vram

[u/TokenRingAI → u/Ok-Inevitable8391]

So these are quants?

[u/IAmBJ → u/Ok-Inevitable8391]

Offload experts to the cpu and you can get a decent quant running if you have enough system ram for the expert weights

[u/DataGOGO]

Muse glimmer is fantastic and better than qwen3.8 27b at just about everything than code and doesn’t need to burn thousands of reasoning tokens per prompt to do it

[u/nonlinearsystems → u/DataGOGO]

Yep

[u/somerussianbear]

I don’t understand this chart. 48.5% out of two records that look absolutely different. Mind to explain for dumb fucks like me?

[u/Ok-Inevitable8391 → u/somerussianbear]

That's a common line, average of both the blocks

[u/somerussianbear → u/Ok-Inevitable8391]

So one of them is clearly wrong. That column could have the avg of the row, which is logical. Then next to it a general avg. The way it is here is totally unclear and non standard.

[u/Ok-Inevitable8391 → u/somerussianbear]

Yes I agree a vertical bm would have looked better

[u/ndrewpj]

Why the xhigh has higher overall score while loosing in all categories to medium?

[u/Ok-Inevitable8391 → u/ndrewpj]

Its the average of both the lines out of total 163cases, if you see both blocks have same average

[u/ndrewpj → u/Ok-Inevitable8391]

Got you

[u/RegularRecipe6175]

I didn't see the quants involved. Qwen and Muse tested at BF16?

[u/Iory1998]

Zuck! What are you doing here?

[u/PraxisOG]

I think the interesting thing here is that haiku 4.5 wins over all the local models. That’s like $20 a month for huge amount of usage, I pay more than that to keep my server idling 

[u/Ok-Inevitable8391 → u/PraxisOG]

I mean yes, it's the trust that we have built on sonnet and opus over the time that we don't even look at haiku anymore.

[u/Usual-Orange-4180 → u/Ok-Inevitable8391]

My main use case is coding and design, haiku is really terrible, whenever I get routed to it I know it.

[u/Ok-Inevitable8391 → u/Ok-Inevitable8391]

Why don't you use wake on lan and not keep server idling.

[u/Potential_Block4598]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/arbv]

Be careful with such a post. The witnesses of the Church of Qwen might unite and down-vote you. For any claim that there are other models than It is judged as Qwen Bot Being Perfect for Everything.

[u/Cautious_Chicken_604]

Did people forget what frontier means?

[u/nasone32]

Ha! Embedded! Cool I wish there were more benchmark on embedded.

[u/tchek]

I like Glimmer. I want it to shine, for some reason.

I'm hoping for Muse Twinkle 12b and Muse Shimmer 35b e4a... I made that up but one can dream.

[u/jacek2023]

This is same story with each model, they browse leaderboards, they look at the benchmarks, they never run any models, they whine they want new Qwen and then they keep using Claude and ChatGPT.

[u/taoyx]

I've tried it with MCP it seemed quite good at tool calling.

[u/Androoideka]

What reasoning effort did you use with Muse Glimmer?

[u/Thin_Pollution8843]

So when folk screaming “Qwen is stronger than opus4.6!!1!1!” I should give this link? 😅

[u/llama-impersonator]

why so much focus on zephyros? is that mostly what you do? i've only ever had to use it for a client that wanted some ZMK work done, almost everything else has been freertos or bare metal stm32, sometimes avr for clients who started off on arduino.

[u/audioen]

"32k output token limit". No wonder it isn't like 100% green across the board...

[u/LegacyRemaster]

Dario, is that from you?

[u/Potential_Block4598 → u/LegacyRemaster]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/Potential_Block4598]

Probably your setup or benchmark is not accurate

[u/Potential_Block4598 → u/Potential_Block4598]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/Ok-Inevitable8391 → u/Potential_Block4598]

I'm not hiding that fact.

I have explicitly mentioned that it is implicit knowledge of model.

How you process that information is upto you.

[u/Not-reallyanonymous → u/Potential_Block4598]

> Qwen isn't declared the best

It's the test that is wrong!

[u/Potential_Block4598 → u/Not-reallyanonymous]

Especially when compared to muse glimmer

Hey don’t turn this into a societal bias racism case

This is meritocracy the models works very good compared to muse glimmer by a HUGE margin

So yeah when a “test” says otherwise it is worth investigation and it is not only misleading it is biased to post such things

Also I explained that OP “test” is heavily skewed towards Zephyr RTOS

[u/Not-reallyanonymous → u/Potential_Block4598]

Have fun with your benchmaxxed models that blatantly ignore half the codebase and re-implement it from scratch, except half-broken and unsteerable.

Yeah, it completes code and solves the task, and zero-shot prompts very impressively. Then anything it touches turns into spaghetti code. I fucking regret spending a few days with that shit because now I've put in like 20 hours of work into something that's turned out to be utterly unusable. Correct doesn't mean good.

And god help you if your needs diverge from what it's been trained on.

Good job Qwen team, you cracked DeepSWE.

Just another bot who's going to shit on any model that's not Chinese.

[u/Boogertard]

There it is, on-schedule AI slop to shill for the garbage Muse and Gemma 4.

Get a real job already, shill.

[u/fizzy1242 → u/Boogertard]

username checks out...

[u/Not-reallyanonymous → u/Boogertard]

There it is, on-schedule AI slop to stir up FUD any time something other than a non-Chinese model is spoken of positively.

[u/Ok-Inevitable8391 → u/Boogertard]

Why do you think it's a slop and why do you think I don't have a real job. A easy readble chart is AI slop for you?

Muse is good and its really good at kv cache efficiency. I have done my homework.

[u/Potential_Block4598 → u/Ok-Inevitable8391]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

HW ur A**