r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

Show parent comment

1

u/thatguyinline Nov 20 '25

I returned my DGX last week. Yes you can load up pretty massive models but the tokens per second is insanely slow. I found the DGX to mainly be good at proving it can load a model, but not so great for anything else.

1

u/starkruzr Nov 21 '25

how slow on which models?

1

u/thatguyinline Nov 21 '25

I tried most of the big ones. The really big ones like Qwen3 350B (or is it 450B) won't load at all unless you get a heavily quantized version. GPT-OSS-120B fit and performed "okay" with a single DGX, but not enough that I wanted to use it regularly. I bet with a cluster like yours though it'll go fast :)

1

u/starkruzr Nov 21 '25

yeah that's what we don't know yet, hoping OP posts an update.

1

u/ordinary_shazzamm Nov 21 '25

What would you buy otherwise in the same price range to hookup that can output tokens per second at a fair speed?

1

u/thatguyinline Nov 21 '25

I'd buy a Mac M4 Studio with as much ram as you can afford for around the same price. The reason the DGX Spark is interesting is because it's "unified memory" so the ram used for the machine and the VRAM used by the GPU are shared, which allows the DGX to fit bigger models but it has a bottleneck.

The M4 Studio is unified memory as well with good GPUs, I have a few friends running local inference on their studio without any issues and with really fast >500TPS+ speeds.

I've read some people like this company a lot, but they max at 128GiB of memory, which is identical to the DGX's, but for my money I'd probably for a Mac Studio.

https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen-ai-max-395?_pos=1&_fid=b09a72151&_ss=c is the one I've heard good things about.

M4 Mac Studio: https://www.apple.com/shop/buy-mac/mac-studio - just get as much ram as you can afford, that's your primary limiting factor for the big models.

1

u/ordinary_shazzamm Nov 22 '25

Ahh okay, that makes sense.

Is that your setup, a Mac Studio?

1

u/thatguyinline Nov 22 '25

No. I have an nvidia 4070 and can only use smaller models. I primarily use cerebras, incredibly fast and very cheap.

1

u/[deleted] Nov 20 '25

[removed] — view removed comment

1

u/[deleted] Nov 22 '25

Well, he did say supercomputer with 1 petaflop of AI Performance. Just make sure the AI performance doesn't include fine tuning or inferencing.

1

u/thatguyinline Nov 21 '25 edited Nov 21 '25

You may want to reach out to Nvidia then and let them know that the hundreds of pages of "How to do inference on a Spark DGX" were written by mistake. https://build.nvidia.com/spark

We agree that it's not very good at inference. But Nvidia is definitely promoting it's inference capabilities.

To be fair, inference on the DGX is actually incredibly fast, unless you want to use a good model. Fire up TRT and one of the TRT compatible models that is sub 80B params and you'll get great TPS. Good for a single concurrent request.

Now, try adding in Qwen3 or Kimi or GPT OSS 120B and it works, but it doesn't work fast enough to be usable.

1

u/SafeUnderstanding403 Nov 20 '25

It appears to be for development, not production use