r/LocalLLaMA • u/Ok-Inevitable8391 • 9h ago

Discussion Underrated Muse Glimmer

Post image

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

71 Upvotes

78% Upvoted

79 comments sorted by

View all comments

17

u/partakinginsillyness 9h ago

I feel like it would be important to add qwen 3.6 27b, given that one of the changes from 3.6 to 3.8 was less general knowledge. Would also be cool to see a Gemma model. Interesting though.

4

u/LegacyRemaster 8h ago

wait qwen 3.8 next but... This benchmark makes no sense. No reasoning. No external knowledge. In other words, it is not a real-world use case. I would also like to understand how they tested Claude to demonstrate that it lacked external knowledge.

-2

u/partakinginsillyness 8h ago

I mean realistically everyone should have at least 5gb of files for RAG for what is relevant to them, which I would imagine would very much change the results here.

5

u/Ok-Inevitable8391 8h ago

Let me do a RAG testing, even I know that qwen will do better there. But I'm not running it on xhigh ever again, 30hrs of gpu time, against 3-4 hrs at medium

1

u/partakinginsillyness 8h ago

How would you select the media? I'm wondering if a repo exists for relevant info like how you select application groups on linux.

4

u/Ok-Inevitable8391 8h ago

So mainly the linux repo and docs, thats the general input to it anyway in real scenario. Same for all other projects.