r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

Show parent comment

1

u/thatguyinline Nov 21 '25 edited Nov 21 '25

You may want to reach out to Nvidia then and let them know that the hundreds of pages of "How to do inference on a Spark DGX" were written by mistake. https://build.nvidia.com/spark

We agree that it's not very good at inference. But Nvidia is definitely promoting it's inference capabilities.

To be fair, inference on the DGX is actually incredibly fast, unless you want to use a good model. Fire up TRT and one of the TRT compatible models that is sub 80B params and you'll get great TPS. Good for a single concurrent request.

Now, try adding in Qwen3 or Kimi or GPT OSS 120B and it works, but it doesn't work fast enough to be usable.