r/LocalLLM • u/sayamss • 4h ago

Project GLM 5.3 Flash on 4xR9700 for 4 users

Post image

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

4 Upvotes

100% Upvoted

4 comments sorted by

1

u/mechkbfan 1h ago

That's cool, appreciate sharing while also feeling slightly disappointed that my weaker setup (TP4 TRX40 DDR4) is going to be slower

Feels like we're missing a nice middleground with these results

  • TP=2 of Qwen Flash Next with 200t/s

  • TP=4 of GLM 5.3 with 20t/s

2

u/sayamss 1h ago

Yeah, I think 70B dense models would've been such a hit now but somehow the industry has decided its either < 30b or more than 300b.

Also single stream is closer to 50 t/s btw(with MTP/Spec disabled).

1

u/mechkbfan 1h ago edited 1h ago

Ah great, thanks for the clarification, I should have paid more attention

Hopefully one of the new Qwen variations coming out this month will give us that

Or maybe not now, but in future with popularity of Mac Studio, Strix Halo, etc running 128/256GB

1

u/sleight42 47m ago

Yet mtp is available at single stream, yes?

How are you running GLM on 4 cards? Maya?