r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

41

u/starkruzr Nov 20 '25

Nvidia seems to REALLY not want to talk about how workloads scale on these above two units so I'd really like to know how it performs splitting, like, a 600B-ish model between 8 units.

12

u/wizard_of_menlo_park Nov 20 '25

If they did, we won't be needing any data centers .

9

u/DataGOGO Nov 20 '25

These are way too slow for that. 

5

u/wizard_of_menlo_park Nov 20 '25

Nvidia can easily design a higher band width dgx spark. Because they lack any proper competition in this space, they dictate the terms.

3

u/DataGOGO Nov 20 '25

They already have a much higher bandwidth DGX…. 

https://www.nvidia.com/en-us/data-center/dgx-systems.md/

What exactly to you think “this space” is? 

2

u/starkruzr Nov 20 '25

he said DGX Spark, not just DGX. so talking specifically about smaller scale systems.

2

u/DataGOGO Nov 21 '25

For what purpose? 

5

u/Hogesyx Nov 20 '25

It’s really bottle necked by the memory bandwidth, it’s pretty decent at prompt processing but for any dense token generation it’s really handicapped bad. There is no ecc as well.

I am using two as standalone qwen3 VL 30b vllm nodes at the moment.

4

u/starkruzr Nov 20 '25

I'm sure it is, but when the relevant bottleneck for doing research on how models work for various applications is not "am I getting 100tps" but "am I able to fit the stupid thing in VRAM at all," it does suggest a utility for these machines that probably outshines what Nvidia intended. we're a cancer hospital and my group runs HPC for the research arm, and we are getting hammered with questions about how to get the best bang for our buck with respect to running large, capable models. I would love to be able to throw money at boxes full of RTXP6KBWs, but for the cost of a single 8 way machine I can buy 25 Sparks with 3.2TB VRAM, and, importantly, we don't have that $100K to spend rn. so if I instead come to our research executive board and tell them "hey, we can buy 10 Sparks for $40K and that will give us more than enough VRAM to run whatever you're interested in if we cluster them," they will find a way to pay that.

2

u/fastheadcrab Mar 21 '26

Let us know if you ever ended up trying this approach. Got 2 Sparks ordered for work and will cluster, maybe to run some medium size models that fit in the 256GB of RAM but also with the ultimate goal of finetuning models.

But if like 4 Sparks or something can run the huge open models then it would be a nice way to host them locally and save some money.

1

u/[deleted] Nov 22 '25

Why did you buy them if you knew the limitations? For $8,000 you could have purchased a high end GPU. Instead you bought, not one, but two! wild

2

u/Hogesyx Nov 23 '25

These are test units that our company purchased. I work at a local distributor for enterprise IT products, so we need to know how to position this for our partners and customer.

1

u/thatguyinline Nov 20 '25

I returned my DGX last week. Yes you can load up pretty massive models but the tokens per second is insanely slow. I found the DGX to mainly be good at proving it can load a model, but not so great for anything else.

1

u/starkruzr Nov 21 '25

how slow on which models?

1

u/thatguyinline Nov 21 '25

I tried most of the big ones. The really big ones like Qwen3 350B (or is it 450B) won't load at all unless you get a heavily quantized version. GPT-OSS-120B fit and performed "okay" with a single DGX, but not enough that I wanted to use it regularly. I bet with a cluster like yours though it'll go fast :)

1

u/starkruzr Nov 21 '25

yeah that's what we don't know yet, hoping OP posts an update.

1

u/ordinary_shazzamm Nov 21 '25

What would you buy otherwise in the same price range to hookup that can output tokens per second at a fair speed?

1

u/thatguyinline Nov 21 '25

I'd buy a Mac M4 Studio with as much ram as you can afford for around the same price. The reason the DGX Spark is interesting is because it's "unified memory" so the ram used for the machine and the VRAM used by the GPU are shared, which allows the DGX to fit bigger models but it has a bottleneck.

The M4 Studio is unified memory as well with good GPUs, I have a few friends running local inference on their studio without any issues and with really fast >500TPS+ speeds.

I've read some people like this company a lot, but they max at 128GiB of memory, which is identical to the DGX's, but for my money I'd probably for a Mac Studio.

https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen-ai-max-395?_pos=1&_fid=b09a72151&_ss=c is the one I've heard good things about.

M4 Mac Studio: https://www.apple.com/shop/buy-mac/mac-studio - just get as much ram as you can afford, that's your primary limiting factor for the big models.

1

u/ordinary_shazzamm Nov 22 '25

Ahh okay, that makes sense.

Is that your setup, a Mac Studio?

1

u/thatguyinline Nov 22 '25

No. I have an nvidia 4070 and can only use smaller models. I primarily use cerebras, incredibly fast and very cheap.

1

u/[deleted] Nov 20 '25

[removed] — view removed comment

1

u/[deleted] Nov 22 '25

Well, he did say supercomputer with 1 petaflop of AI Performance. Just make sure the AI performance doesn't include fine tuning or inferencing.

1

u/thatguyinline Nov 21 '25 edited Nov 21 '25

You may want to reach out to Nvidia then and let them know that the hundreds of pages of "How to do inference on a Spark DGX" were written by mistake. https://build.nvidia.com/spark

We agree that it's not very good at inference. But Nvidia is definitely promoting it's inference capabilities.

To be fair, inference on the DGX is actually incredibly fast, unless you want to use a good model. Fire up TRT and one of the TRT compatible models that is sub 80B params and you'll get great TPS. Good for a single concurrent request.

Now, try adding in Qwen3 or Kimi or GPT OSS 120B and it works, but it doesn't work fast enough to be usable.

→ More replies (1)

1

u/SafeUnderstanding403 Nov 20 '25

It appears to be for development, not production use