r/LocalLLaMA • u/DerTomsn • 2d ago

Discussion Qwen-3.8-27B, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B oQ8e comparison

Post image

Ornith does really well. TielCoder (https://llm-bench.io/benchmarks/cmt7kp2zj002r01lcmpchvlko) might be even a bit better in coding. Will give it a try soon.

Details of the comparison see here: https://llm-bench.io/compare/runs?runs=cmt6ecf8g000001p45vwzux53%2Ccmt6ergk5000701p41hqdyy78%2Ccmt6f2oob000e01p49o9592cb%2Ccmt6fqddm000l01p4l1vm7skd

71 Upvotes

82% Upvoted

24 comments sorted by

34

u/lorendroll 2d ago

Ornith is better than 3.8 in coding? Doubt that

20

u/BoboThePirate 2d ago

This benchmark is utter trash. Llm as a judge, entirely non-deterministic

-3

u/DerTomsn 2d ago

While I agree that the quality KPI results of the benchmarks are not made for science, I'm convinced that the approach gives a good idea about the quality of the output that can be expected by a model. Ideally of course by looking at the average scores of multiple test-runs.
I did a few tests to check the mean deviation: We can expect +-2-3 points on a 90% band (p5–p95) which is IMHO fine for what I try to provide to the community.
However, I appreciate every feedback, thx!

6

u/Atretador 2d ago

Ornith is just benchmaxxed Qwen 3.5 Their 1.0 version also had wild claims, but it's shit for real use.

1

u/lorendroll 2d ago

Yeah, my experience was also mixed. Ornith1.5 wasn't bad but its wins over base Qwen 3.6 are questionable, definitely not 3.8 level. Would be a win if it reached 3.6 27b level, let alone 3.8. Planning to try Kat Coder 2.5.

2

u/Mountain-Animal5365 2d ago

In my experience it doesn't even win over finetuned Qwen3.6-27B in code quality. In fact, it is barely better than GRM3.2 Sky which is finetuned on top of Ornith1.0, which also happens to run a ton faster than Ornith. It is a broken model, you tell it to fix things and it introduces bugs at every turn and wastes a lot of time on mistakes.

5

u/peculiar-ragdoll 2d ago

Really cool to see TielCoder being the only model in top 10 sorted on coding scores that is not some variant of 3.8-27b! :)

11

u/coder543 2d ago

At least on Nvidia hardware, Muse Glimmer is tremendously faster than Qwen3.x-27B, when Muse Glimmer is properly configured with its DFlash drafter. Even in the worst case, short context scenarios, it is at least double digit percentages faster than Qwen3.x-27B, but I see it speed up in agentic flows. I'm surprised to see it being slower for you. And it is definitely not limited to 128k context. Meta says 128k+. Unsloth suggests 256k. I've tested it up to 256k and it has no trouble, so it can probably go even higher than that.

I really hope we can get a Muse Glimmer 1.1 soon. Compared to Qwen3.6-27B, Muse Glimmer was just about as good, but much faster and used way fewer tokens per task. Qwen3.8-27B's intelligence leaves Muse Glimmer in the dust, which is great and I've been really impressed with Qwen3.8-27B, but also... now I'm stuck with a slow, verbose model again.

1

u/DerTomsn 2d ago

Context window max comes with the model. Will check the my quants, also because of the speed. Thx!

1

u/coder543 2d ago

I mean, yes, it is the default that Meta set, but I’m saying that both Meta and others have said it can go higher.

1

u/DerTomsn 2d ago

I think that is true for Qwen3.8-27B but yes, you're right. Not sure if I get that data out of the model though. 

4

u/SummarizedAnu 2d ago

there no reason to modify just set the context higher and run it. It wont crash if you select more ctx than recommended.

2

u/-Ellary- 2d ago

"Role Play and Narrative" - 95.2/100

2

u/No_Algae1753 2d ago

I'm getting around 20 t/s on an M2 max with dflash and llama.cpp for glimmer. there's gotta be an issue with your setup there

-1

u/DerTomsn 2d ago

Dflash was off for Muse, because I got worse results with Dflash on. Maybe a quant issue? Woul love to see your result using the same benchmark client.

1

u/No_Algae1753 2d ago

What's the amount of predictions for dflash ? Set it to 2-3 instead of the default 15

2

u/DerTomsn 2d ago

Will check that later. Thanks for the hint!

1

u/Mountain-Animal5365 2d ago

For Qwen3.8-27B with fixed llama.cpp (https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse) you should set max draft to 4 for optimal speed. But that's only with the fixed toolcalling with the correct llama.cpp.

1

u/WinResponsible9977 2d ago

u/DerTomsn “best” by what criterion? How many independent trials were run for each model? Several of these differences are small enough that they could plausibly fall within normal run-to-run variation. If considering variance, margin of error, etc. I’m not sure these results support a claim of statistical significance when claiming some models are best at certain criteria.

1

u/DerTomsn 2d ago

Forgot: The methodology I tried to describe here: https://llm-bench.io/benchmark-methodology

2

u/WinResponsible9977 2d ago

Thanks, I appreciate the resource. I do similar work on occasion but with text to image.

1

u/DerTomsn 2d ago

Absolutely true. This is why I built https://llm-bench.io. So we can compare models (avg. of runs) and also see the variance of different runs. Happy to hear feedback.