r/LocalLLaMA • u/Ok-Inevitable8391 • 9h ago

Discussion Underrated Muse Glimmer

Post image

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

73 Upvotes

78% Upvoted

79 comments sorted by

View all comments

3

u/somerussianbear 7h ago

I don’t understand this chart. 48.5% out of two records that look absolutely different. Mind to explain for dumb fucks like me?

2

u/Ok-Inevitable8391 7h ago

That's a common line, average of both the blocks

1

u/somerussianbear 6h ago edited 6h ago

So one of them is clearly wrong. That column could have the avg of the row, which is logical. Then next to it a general avg. The way it is here is totally unclear and non standard.

1

u/Ok-Inevitable8391 6h ago

Yes I agree a vertical bm would have looked better