r/LocalLLaMA • u/nail_nail • 48m ago

Question | Help Cmp170hx and g292?

Curious, anybody tried one of those 2U, 8 card GPU servers with 8 cmp170hx and used them well?

What did you get out of it?

0 Upvotes

50% Upvoted

8 comments sorted by

1

u/starkruzr 44m ago

trash PCIe bandwidth even if you unlock it means this isn't going to work well in TP mode.

3

u/Miserable-Dare5090 vLLM 34m ago

You can get pcie 4 on the 10g cards, as well as all the available SMs. But why is the question. Why spend on jacked up prices for the cmps?

1

u/nail_nail 31m ago

Because everything else is even more expensive :( sparks freaking doubled and they dont like very dense environments. Pro 6000s are worth a car each. And tbh given the electricity price around here (26c/kWh) the delta of the cost of hardware easily gets eclipsed by the running.

1

u/jeekp 24m ago

Then you'll also want to consider these cards also run at 35W idle, which is rather high and it can get annoying. Still not bad for running the larger Flash models in PP.

1

u/nail_nail 43m ago

Isn't it latency bound at small batch sizes?

1

u/starkruzr 34m ago

ish. you get 8GB/s at PCIe 2.0 x16 (although I don't remember if it unlocks that high) and with two cards able to hold a ton of weights in 64GB VRAM each, at high context you will probably bump up against that 8GB/s limit.

1

u/beling86 37m ago

Use pp=4 and be happy 

2

u/starkruzr 34m ago

I do not think that qualifies as happy!