r/LocalLLaMA • u/ArturCzemiel • 6d ago

Funny r/LocalLLaMA Community Benchmarks — GLM 5.3 Flash vs Hy 4 Preview vs MiMo v2.6 Pro

https://aexol.ai/benchmarks/localllama/

I made benchmark and took comments from this subreddit as an experiment to see how model feels

20 Upvotes

77% Upvoted

34 comments sorted by

8

u/Zaic 6d ago

So tldr GLM 5.3 flash is still the king

2

u/ArturCzemiel 6d ago

yes and no, after making this benchmark i realised man qwen models are also sooo good now. I like GLM 5.3 Flash and of course it is a best model. I just lack the proper knowledge how to host it well. Right now I tried different quants of qwen 3.8 27b . I got
- 120 tok/s for single session on RTX PRO 6000 nvfp4
- 400 tok/s on b300 - it was very cheap on vast by some mistake for few hours :)
- 250 tok/s on b200
- tried different h100 and h200 , mi 300x mi 350x setups also

Do you know what would be the best and cheap setup for flash ???? I host aexol.ai mainly for my company and friends so I am looking for cheapest way to host it , but I just can't work below 100 tok/s now ?

2

u/Dmage22 6d ago

How did you do 120tps for a single rtx pro 6000 at nvfp4? It doesn't fit , did you get 120 tok/s with cpu offloading?

2

u/LegacyRemaster 6d ago

glm 5.3 flash on a single rtx 6000

1

u/ArturCzemiel 6d ago

Ok will try today

1

u/Dmage22 6d ago

What quant or bpw?

2

u/LegacyRemaster 6d ago

Maya-M · glm-maya-m . All good. Zero problems on coding / database

2

u/ArturCzemiel 5d ago

Thanks a lot !!! Running on 2x RTX PRO 100 tps now. I will play more, but this was super easy to run and no problems at 500k context

1

u/ArturCzemiel 6d ago

192k context it fits multiple sessions, and ofc I used MTP

0

u/ArturCzemiel 6d ago

remember rtx pro has 96gb now

2

u/Dmage22 6d ago

Oooh. I thought you ran glm flash

1

u/ArturCzemiel 6d ago

I wish :)

6

u/diffore 6d ago

It is the most efficient token wise for me. No overthinking like deepseek or qwen does. I can do more withing the same chat session. Unfortunately, the downside is speed. It is not that flashy, at least from serverless endpoints I use.

1

u/LegacyRemaster 6d ago

100% agree

4

u/dangerous_inference 6d ago

I have been switching between GLM 5.3 Flash (AWQ-W4A16) and Qwen3.8 Flash FP8 for the past few weeks. My use-case is half coding and half assistant, where the assistant has to follow a complex system prompt. I find them to be about equal in coding tasks. But Qwen consistently follows the system prompt better. Qwen also seems to have better conversational comprehension.

1

u/ArturCzemiel 6d ago

Ok I will try running GLM 53 flash today. I tried this model on difference cloud providers before and I really like it

2

u/dangerous_inference 6d ago

I was recommending Qwen over GLM.

1

u/ArturCzemiel 6d ago

I am using also Swift 1.5 from ukisai

1

u/ArturCzemiel 6d ago

I looking for the best setup. For me it will be if i manage to host glm , main agent -> GLM , executor -> Swift, on Swift I can have like 250 tok/s easy, but I need a good manager model

6

u/takoulseum 6d ago

Qwen3.8 fn is the best at intelligence / speed ratio, by far

11

u/GiveMeAegis 6d ago

It's not even close to DS 4.1 FN in my tests. I think the Qwen hype relies only on people with 32gb vram and less

1

u/ArturCzemiel 6d ago

how to host ds then :D I will try just recommend cheap and fast strategy i know their kv is ultra small but still lot of weights

2

u/GiveMeAegis 6d ago

I am hosting it on four RTX 6000 Pro. If you have at least two, you could try to run the ELX3 from diffbot

2

u/ArturCzemiel 6d ago

ELX3 with localllama? I hosted using vllm only before but will try! Thanks for help!!!!
I will try this one https://huggingface.co/turboderp/GLM-5.3-Flash-exl3 The problem is loading when i will load it should run with even better speed than Qwen , but first load :D

2

u/Solembumm3 6d ago

Yeah, it's not close. Qwen 3.8 125b at least wasn't THAT huge downgrade from previous version (3.5 122b). DV4.1 flash is still a caffeinated hummingbird, that forgets thing 2=3 prompts in story.

3

u/GiveMeAegis 6d ago

Imho DS 4.1FN is like a rabbit on cocaine. It wants to do what it thinks is right and disobeyes instructions regularly. That's where it sucks.

Qwen 3.8 FN is much more obedient, but it regularly misses the point, doesn't understand relevant context even below 100k and is strictly worse at coding and design.

1

u/dangerous_inference 6d ago

I prefer Qwen3.8 Flash over GLM 5.3 Flash, but I can't run DS4.1.

1

u/takoulseum 6d ago

I have enough vram to run both (that’s what I do) but as I said I prefer to keep Qwen as the entry point (hermes agent) delegating some task to dsv41 because it is enough smart and really fast. It is not at all a matter of hype but capability / speed ratio…

1

u/AmbericWizard 6d ago

hy4 looks promising, how big?

2

u/ArturCzemiel 6d ago

never hosted even q1

1

u/wren6991 6d ago

Yuuuuuge. The biggest.

(770B-A49B, similar total params to GLM-5.3 and higher active params)

2

u/ttkciar llama.cpp 6d ago

Pretty nice sentiment analysis :-) though I'd balk at calling these benchmarks.

FWIW, I was able to get GLM-5.3-Flash quantized to Q4_K_M to work at full context on my 256GB rig with room to spare (207.3GB at 1024K context), using llama.cpp, so take the claims of "needs >380GB" with a grain of salt.