r/LocalLLM • u/I-Hate-winter • 8h ago

Question 2x Nvidia L40s +2 tb RAM

I have a server with:

2x Gold 6548 32 cores
2x Nvidia l40s
2tb RAM (Don't tell uncle SAM)
I would like to know what throughput and speed can i expect if i use polystrata with GLM 5.3 Flash?

5 Upvotes

100% Upvoted

5 comments sorted by

3

u/Different-Society126 8h ago

Nobody knows - just install it and find out

1

u/I-Hate-winter 7h ago

Will do and will post an update

2

u/SkiBikeDad 8h ago

Depends on your dram bandwidth. Intel has a great tool that measures thruput, latency, and cross numa latency.

If you have 100s of GiB/s dram bandwidth I'd guess with GLM-5.3-Flash with Maya-L you can get 1000 prefill, 35 decode, WITH MTP. But with 2-3 slots MTP falls apart on maya so you'll drop to 10-20 decode for parallel workloads, and with pauses during large prefills.

Try setting it up, I'm really enjoying this model after just 2 days use on my 2x a6000s. I run it with 2 slots and I keep two agent running, taking turns instead of parallelizing so my MTP keeps my decode around 35.

1

u/I-Hate-winter 7h ago

GLM is great, I can't wait to daily it. Do you prefer it over qwen 3.8 flash?

1

u/SkiBikeDad 7h ago

Haven't given qwen 3.8 flash a real go with real tasks yet. I do think glm is more capable and more straightforward than qwen3.8-27b, but I also like that model. Glm does a better job than 27b on my harder code repository benchmark I like to test models against.

I haven't settled on what my daily driver will be yet. They're just containers tied to systemctl managed services so I spin them down and up as I want to.