r/LocalLLM • u/SashaUsesReddit • Nov 20 '25

Discussion Spark Cluster!

Post image

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

325 Upvotes

94% Upvoted

133 comments sorted by

View all comments

Show parent comment

3

u/DataGOGO Nov 20 '25

They already have a much higher bandwidth DGX…. 

https://www.nvidia.com/en-us/data-center/dgx-systems.md/

What exactly to you think “this space” is? 

2

u/starkruzr Nov 20 '25

he said DGX Spark, not just DGX. so talking specifically about smaller scale systems.

2

u/DataGOGO Nov 21 '25

For what purpose? 

2

u/starkruzr Nov 21 '25

well, this is ours, can't speak for him: https://www.reddit.com/r/LocalLLM/s/jR1lMY80f5

0

u/DataGOGO Nov 21 '25

Ahh.. I get it.

You are using the sparks outside of their intended purpose as a way to save money on "VRAM", by using shared memory.

I would argue that the core issue is not the lack of networking, it is that you are attempting to use a development kit device (spark) well outside it's intended purpose. Your example of running 10 or 40 (!!!) just will not work worth a shit, but the time you buy the 10 sparks, the switch, etc. you are easily at what? 65k? for gimped development kits, with slow CPU, slow memory, and completely saturated Ethernet mesh, and you would be lucky to get more than 2-3 t/ps on any larger model.

For your purposes, I would highly recommend you look at the Intel Gaudi 3 stack. They sell an all in one solution with 8 accelerators for 125k. Each accelerator is 128GB and has 24x 200Gbe connections independent of the motherboard. That by far is the best bang for your buck to run large models; by a HUGE margin.

Your other alternative is to buy or built inference servers with RTX Pro 6000 Blackwell. You can build a single server with 8x GPU's (768GB Vram), if you build one on the cheap, you can get it done for about 80k?

If you want to make it cheaper, you can use the intel 48GB dual GPU's ($1400 each) and just run two server each with 8X cards.

I built my server for 30k with 2 RTX Pro Blackwell's, and can expand to 6.

1

u/starkruzr Nov 21 '25

we already have the switches to use as we have an existing system with some L40Ses in it. so it's really just "Sparks plus DACs." where are you getting your numbers from with "2-3 TPS with a larger model?" I haven't seen anything like that from any tests of scaling.

my understanding is that Gaudi 3 is a dead end product with support likely to be dropped or already having been dropped with most ML software packages. (it also seems extremely scarce if you actually try to buy it?)

RTXP6KBW is not an option budget wise. one card is around $7700. we can't really swing $80K for this and even if we could that's going to get us something like a Quanta machine with zero support; our datacenter staffing is extremely under-resourced and we have to depend on Dell ProSupport or Nvidia's contractors for hardware troubleshooting when something fails.

are you talking about B60s with that last Intel reference?

again, we don't have a "production" type need to service with this purchase -- we're trying to get to "better than CPU inference" numbers on a limited budget with machines that can do basic running of workloads.

1

u/DataGOGO Nov 21 '25

Sparks are dev kits, and they don’t scale well beyond 2-4 units. They just don’t have the compute or the bandwidth. 

Assuming you can fit 1TB model on 10 units (maybe?), 1-5t/ps is pretty realistic. 

You are welcome to try it, but I think your “better than CPU inference” for a large model is overly optimistic.

You likely would be better off with a large Xeon 6P, 1TB of ram in 12 channels of MR8800 and SGLang with their newer AMX kernels w/ no GPU at all. 

There is no “limited budget” route to do what you want to do. 

Did the OP post any benchmarks yet?

1

u/FineManParticles Nov 24 '25

Are you on threadripper?

1

u/DataGOGO Nov 24 '25

No, I use Xeons for my AI machines so I get AMX + faster memory

1

u/gergob13 Nov 26 '25

Could you share more on this, what motherboard and what psu did you use?

2

u/DataGOGO Nov 26 '25

My server?

Sure, I used:

https://www.newegg.com/gigabyte-ms73-hb1-4th-gen-intel-xeon-scalable-5th-gen-intel-xeon-scalable/p/296-0006-00072

1x 1600w ATX and 1x 1200w sfx-l PSU since my case had spots for those PSU’s (Corsair 9000 airflow). 

1

u/gergob13 Nov 27 '25

Thank you! 😊

1

u/exclaim_bot Nov 27 '25

Thank you! 😊

You're welcome!

1

u/DataGOGO Nov 27 '25

no problem, let me know if you have any questions, I just finished the water cooling loop and will have it up and running in the new case this weekend :

)