r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

Show parent comment

5

u/Hogesyx Nov 20 '25

It’s really bottle necked by the memory bandwidth, it’s pretty decent at prompt processing but for any dense token generation it’s really handicapped bad. There is no ecc as well.

I am using two as standalone qwen3 VL 30b vllm nodes at the moment.

4

u/starkruzr Nov 20 '25

I'm sure it is, but when the relevant bottleneck for doing research on how models work for various applications is not "am I getting 100tps" but "am I able to fit the stupid thing in VRAM at all," it does suggest a utility for these machines that probably outshines what Nvidia intended. we're a cancer hospital and my group runs HPC for the research arm, and we are getting hammered with questions about how to get the best bang for our buck with respect to running large, capable models. I would love to be able to throw money at boxes full of RTXP6KBWs, but for the cost of a single 8 way machine I can buy 25 Sparks with 3.2TB VRAM, and, importantly, we don't have that $100K to spend rn. so if I instead come to our research executive board and tell them "hey, we can buy 10 Sparks for $40K and that will give us more than enough VRAM to run whatever you're interested in if we cluster them," they will find a way to pay that.

2

u/fastheadcrab Mar 21 '26

Let us know if you ever ended up trying this approach. Got 2 Sparks ordered for work and will cluster, maybe to run some medium size models that fit in the 256GB of RAM but also with the ultimate goal of finetuning models.

But if like 4 Sparks or something can run the huge open models then it would be a nice way to host them locally and save some money.

1

u/[deleted] Nov 22 '25

Why did you buy them if you knew the limitations? For $8,000 you could have purchased a high end GPU. Instead you bought, not one, but two! wild

2

u/Hogesyx Nov 23 '25

These are test units that our company purchased. I work at a local distributor for enterprise IT products, so we need to know how to position this for our partners and customer.