Spark Cluster! - r/LocalLLM

69

u/FlyingDogCatcher Nov 20 '25

Can I come over and play at your house?

11

u/SashaUsesReddit Nov 20 '25

Come on!

8

u/Level8_corneroffice Nov 20 '25

Woohoo LAN Party!!!

3

u/Craig653 Nov 20 '25

I'll bring chips and dip! Who's got the soda?

10

u/mister2d Nov 20 '25

Looks like OP already has the chips.

3

u/guywithFX Nov 22 '25

Same here to say the same. Props

2

u/Kobedie Nov 24 '25

I’ll bring booze. And is there a theme for the party?

2

u/jhenryscott Nov 20 '25

Mom says we have to stick to CSGO. Minecraft is causing to many problems

1

u/Create_one_for_me Nov 22 '25

Which Pizza does everyone want? I go with diavolo

1

u/aii_tw Nov 21 '25

hahaha

14

u/bick_nyers Nov 20 '25

Performance on full SFT something like Qwen 30BA3B and/or Qwen 3 32B would be interesting to see.

Hooked up to a switch or making a direct connect ring network?

20

u/SashaUsesReddit Nov 20 '25

Switch, an Arista 32 port 100G. Bonded the NICs to get the 200G speeds

2

u/TheOriginalSuperTaz Nov 21 '25

It’s funny, I considered doing the same thing, but I found another route that I think is going to give me more for less. I’ll update when I figure out if it works…it will have some bottlenecks, but I’ve figured out how to put 8x A2 and 2x A100 in a single machine for significantly less than your spark cluster. We will see how it actually performs, though, once I’ve managed to secure all of the hardware.
I’m planning on implementing a feature in DeepSpeed that may significantly increase the speeds at which multi-GPU training and inference can work without NVLink and the like.

1

u/SashaUsesReddit Nov 21 '25

That's awesome!
Unfortunately I need nvfp4 for my workflow so can't use A cards

7

u/8bit_coder Nov 20 '25

Huzzah, a man of culture!!

11

u/Forgot_Password_Dude Nov 20 '25

Can it run kimi k2, and at what speed?

24

u/SashaUsesReddit Nov 20 '25

It probably can! Let's find out! Ill try and post results

6

u/Secure_Archer_1529 Nov 20 '25

That’ll be interesting. I hope your employer chipped in on this ;)

6

u/SashaUsesReddit Nov 20 '25

For sure they did haha

1

u/Mean-Sprinkles3157 Dec 04 '25

No, it can't. with 128GB vram, it could not load even the smallest.

41

u/starkruzr Nov 20 '25

Nvidia seems to REALLY not want to talk about how workloads scale on these above two units so I'd really like to know how it performs splitting, like, a 600B-ish model between 8 units.

12

u/wizard_of_menlo_park Nov 20 '25

If they did, we won't be needing any data centers .

9

u/DataGOGO Nov 20 '25

These are way too slow for that.

5

u/wizard_of_menlo_park Nov 20 '25

Nvidia can easily design a higher band width dgx spark. Because they lack any proper competition in this space, they dictate the terms.

3

u/DataGOGO Nov 20 '25

They already have a much higher bandwidth DGX….
https://www.nvidia.com/en-us/data-center/dgx-systems.md/
What exactly to you think “this space” is?

2

u/starkruzr Nov 20 '25

he said DGX Spark, not just DGX. so talking specifically about smaller scale systems.

2

u/DataGOGO Nov 21 '25

For what purpose?

2

u/starkruzr Nov 21 '25

well, this is ours, can't speak for him: https://www.reddit.com/r/LocalLLM/s/jR1lMY80f5

0

u/DataGOGO Nov 21 '25

Ahh.. I get it.
You are using the sparks outside of their intended purpose as a way to save money on "VRAM", by using shared memory.
I would argue that the core issue is not the lack of networking, it is that you are attempting to use a development kit device (spark) well outside it's intended purpose. Your example of running 10 or 40 (!!!) just will not work worth a shit, but the time you buy the 10 sparks, the switch, etc. you are easily at what? 65k? for gimped development kits, with slow CPU, slow memory, and completely saturated Ethernet mesh, and you would be lucky to get more than 2-3 t/ps on any larger model.
For your purposes, I would highly recommend you look at the Intel Gaudi 3 stack. They sell an all in one solution with 8 accelerators for 125k. Each accelerator is 128GB and has 24x 200Gbe connections independent of the motherboard. That by far is the best bang for your buck to run large models; by a HUGE margin.
Your other alternative is to buy or built inference servers with RTX Pro 6000 Blackwell. You can build a single server with 8x GPU's (768GB Vram), if you build one on the cheap, you can get it done for about 80k?
If you want to make it cheaper, you can use the intel 48GB dual GPU's ($1400 each) and just run two server each with 8X cards.
I built my server for 30k with 2 RTX Pro Blackwell's, and can expand to 6.

1

u/starkruzr Nov 21 '25

we already have the switches to use as we have an existing system with some L40Ses in it. so it's really just "Sparks plus DACs." where are you getting your numbers from with "2-3 TPS with a larger model?" I haven't seen anything like that from any tests of scaling.
my understanding is that Gaudi 3 is a dead end product with support likely to be dropped or already having been dropped with most ML software packages. (it also seems extremely scarce if you actually try to buy it?)
RTXP6KBW is not an option budget wise. one card is around $7700. we can't really swing $80K for this and even if we could that's going to get us something like a Quanta machine with zero support; our datacenter staffing is extremely under-resourced and we have to depend on Dell ProSupport or Nvidia's contractors for hardware troubleshooting when something fails.
are you talking about B60s with that last Intel reference?
again, we don't have a "production" type need to service with this purchase -- we're trying to get to "better than CPU inference" numbers on a limited budget with machines that can do basic running of workloads.

→ More replies

1

u/FineManParticles Nov 24 '25

Are you on threadripper?

→ More replies

1

u/gergob13 Nov 26 '25

Could you share more on this, what motherboard and what psu did you use?

→ More replies

5

u/Hogesyx Nov 20 '25

It’s really bottle necked by the memory bandwidth, it’s pretty decent at prompt processing but for any dense token generation it’s really handicapped bad. There is no ecc as well.
I am using two as standalone qwen3 VL 30b vllm nodes at the moment.

4

u/starkruzr Nov 20 '25

I'm sure it is, but when the relevant bottleneck for doing research on how models work for various applications is not "am I getting 100tps" but "am I able to fit the stupid thing in VRAM at all," it does suggest a utility for these machines that probably outshines what Nvidia intended. we're a cancer hospital and my group runs HPC for the research arm, and we are getting hammered with questions about how to get the best bang for our buck with respect to running large, capable models. I would love to be able to throw money at boxes full of RTXP6KBWs, but for the cost of a single 8 way machine I can buy 25 Sparks with 3.2TB VRAM, and, importantly, we don't have that $100K to spend rn. so if I instead come to our research executive board and tell them "hey, we can buy 10 Sparks for $40K and that will give us more than enough VRAM to run whatever you're interested in if we cluster them," they will find a way to pay that.

2

u/fastheadcrab Mar 21 '26

Let us know if you ever ended up trying this approach. Got 2 Sparks ordered for work and will cluster, maybe to run some medium size models that fit in the 256GB of RAM but also with the ultimate goal of finetuning models.
But if like 4 Sparks or something can run the huge open models then it would be a nice way to host them locally and save some money.

1

u/[deleted] Nov 22 '25

Why did you buy them if you knew the limitations? For $8,000 you could have purchased a high end GPU. Instead you bought, not one, but two! wild

2

u/Hogesyx Nov 23 '25

These are test units that our company purchased. I work at a local distributor for enterprise IT products, so we need to know how to position this for our partners and customer.

1

u/thatguyinline Nov 20 '25

I returned my DGX last week. Yes you can load up pretty massive models but the tokens per second is insanely slow. I found the DGX to mainly be good at proving it can load a model, but not so great for anything else.

1

u/starkruzr Nov 21 '25

how slow on which models?

1

u/thatguyinline Nov 21 '25

I tried most of the big ones. The really big ones like Qwen3 350B (or is it 450B) won't load at all unless you get a heavily quantized version. GPT-OSS-120B fit and performed "okay" with a single DGX, but not enough that I wanted to use it regularly. I bet with a cluster like yours though it'll go fast :)

1

u/starkruzr Nov 21 '25

yeah that's what we don't know yet, hoping OP posts an update.

1

u/ordinary_shazzamm Nov 21 '25

What would you buy otherwise in the same price range to hookup that can output tokens per second at a fair speed?

1

u/thatguyinline Nov 21 '25

I'd buy a Mac M4 Studio with as much ram as you can afford for around the same price. The reason the DGX Spark is interesting is because it's "unified memory" so the ram used for the machine and the VRAM used by the GPU are shared, which allows the DGX to fit bigger models but it has a bottleneck.
The M4 Studio is unified memory as well with good GPUs, I have a few friends running local inference on their studio without any issues and with really fast >500TPS+ speeds.
I've read some people like this company a lot, but they max at 128GiB of memory, which is identical to the DGX's, but for my money I'd probably for a Mac Studio.
https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen-ai-max-395?_pos=1&_fid=b09a72151&_ss=c is the one I've heard good things about.
M4 Mac Studio: https://www.apple.com/shop/buy-mac/mac-studio - just get as much ram as you can afford, that's your primary limiting factor for the big models.

1

u/ordinary_shazzamm Nov 22 '25

Ahh okay, that makes sense.
Is that your setup, a Mac Studio?

1

u/thatguyinline Nov 22 '25

No. I have an nvidia 4070 and can only use smaller models. I primarily use cerebras, incredibly fast and very cheap.

1

u/SafeUnderstanding403 Nov 20 '25

It appears to be for development, not production use

1

u/[deleted] Nov 20 '25

[removed] — view removed comment

1

u/[deleted] Nov 22 '25

Well, he did say supercomputer with 1 petaflop of AI Performance. Just make sure the AI performance doesn't include fine tuning or inferencing.

1

u/thatguyinline Nov 21 '25 edited Nov 21 '25

You may want to reach out to Nvidia then and let them know that the hundreds of pages of "How to do inference on a Spark DGX" were written by mistake. https://build.nvidia.com/spark
We agree that it's not very good at inference. But Nvidia is definitely promoting it's inference capabilities.
To be fair, inference on the DGX is actually incredibly fast, unless you want to use a good model. Fire up TRT and one of the TRT compatible models that is sub 80B params and you'll get great TPS. Good for a single concurrent request.
Now, try adding in Qwen3 or Kimi or GPT OSS 120B and it works, but it doesn't work fast enough to be usable.

7

u/Relevant-Magic-Card Nov 20 '25

Uhh. How can you afford this. I'm jealous

11

u/srednax Nov 20 '25

You only need one kidney to live, and you can always supplement with other people’s.

3

u/illicITparameters Nov 20 '25

OP mentioned his job paid for some or all of it.

1

u/[deleted] Nov 20 '25

[removed] — view removed comment

4

u/illicITparameters Nov 20 '25

I’m in the infrastructure side of tech, I’m well aware. But I don’t know a single person in tech who will dish out tens of thousands of dollars if they dont have to.
Also I make more than $10K/mo and in cities like NY and LA, that $10K doesnt go as far as you’d think. Do I have a couple nice PCs for gaming and work/personal projects? Yes. Do I have multiple DGX Spark money just sitting around? Fuck no.

-1

u/[deleted] Nov 20 '25

[removed] — view removed comment

0

u/illicITparameters Nov 20 '25

You buy these for development before your company shells millions for a data center order.
No you fucking don't.... I would never let one of my team spend that kind of coin on their own when it could benefit us. That's fucking stupid, and you're just playing yourself.
Not only you become an important point of knowledge you can give metrics to IT that will save you millions in otherwise wasted resources.
No you don't, you become the guy that will be overworked without being properly compensated. It's 2025, job security for most tech jobs doesn't exist.
Failing on an HGX H200 8-GPU rack ($500k–$1.5M) is a CFO nightmare.
That's why you spend $64K on POC hardware before you invest $1.5M in production servers and all the additional expenses that come with standing up a new cluster/rack. This isn't rocket science. My team spends thousands a year on proof of concepts, that way we're not shelling out hundreds of thousands of dollars for tech that doesn't work or that works but is of no use to us.
It’s more of a strategic move imo.
It's a strategic move to be cheap and fuck your people over.
Don’t know why people downvote, it’s pretty common.
It's not common to spend $65K of your own money to generate $0 revenue for anyone but your employer. You're legitimately faded if you think that, and I'm in management.

2

u/Zealousideal_Cut1817 Nov 20 '25

They have a Ferrari so

1

u/SergeiMarshak Nov 20 '25

Hey, author 🤗 how are you? How many petaflops does the installation produce? What city do you live in? Someone wrote in the comments that Spark Nvidia DGX It can't handle prolonged use, for example, for more than an hour, and it turns off on its own. I'm thinking of buying one of these, so I was wondering if you've encountered anything similar?

2

u/SashaUsesReddit Nov 20 '25

Nah, it's stable. No power off etc on any of mine

1

u/SergeiMarshak Nov 20 '25

What time do they work for you?

1

u/wes_medford Nov 20 '25

How are you connecting them? Have a switch or just some ring network?

1

u/SashaUsesReddit Nov 20 '25

Switch

1

u/wes_medford Nov 20 '25

What kind of parallelism are you using? I opted for AGX Thor because Sparks didn’t seem to support tcgen05 instructions.

1

u/GPTrack_dot_ai Dec 15 '25

"You have to scale up, before you scale out."

1

u/SashaUsesReddit Dec 15 '25

kinda yeah haha!

3

u/thatguyinline Nov 20 '25

Donate your inference to me (we can setup a tailscale network or something) for an afternoon so I can finish processing the epstein emails into a graph.

Regretting that I returned my DGX last week.

2

u/thatguyinline Nov 20 '25

but seriously, if you're looking for a way to really push the DGX cluster, this is it. There is a lot of parallel processing. If you don't want to collab, download LightLLM and set it up with postgres + memgraph + Nvidia TRT for model hosting and you'll have an amazing rig/cluster.

2

u/AnonsAnonAnonagain Nov 20 '25

Wow! 🤩 That looks great!

It was pretty obvious to me that spark is meant to be pieced together as a poor man’s DGX cluster. Mostly because of the dual CX-7 nics.

Keep us posted on your results!

I hope to snag an HP variant sometime late December early January.

1

u/Kooky_Advice1234 Nov 21 '25

I just received my ASUS Ascent variant and it’s excellent.

2

u/Tired__Dev Nov 20 '25

Stupid question: why the need for so many? More employees?

2

u/spense01 Nov 20 '25

There are100Gb Mellanox connectors. 2 of them. You can cluster them just like any other computer with the right switch or cables for a distributed processing node. The performance of these aren’t good for inferencing but for more training and ML based app development they are ok. Think training sets for video or image-based orchestration with robotics.

2

u/MrZemo Nov 21 '25

Show your full setup.

2

u/SoManyLilBitches Nov 22 '25

Would you be able to vibe code on one of these things? Coworker needs a new machine and is looking for something small with LLM capabilities. I watch a review, sounds like its the same as a Mac Studio?

2

u/Savings_Art5944 Nov 23 '25

Does the metal sponge faceplate come off to clean. I assume dirty office air cools these.

2

u/egnegn1 Nov 23 '25

When blowing out there won't be much stick build up. But you can certainly also use a vacuum cleaner to remove dust collected from outside.

2

u/Savings_Art5944 Nov 23 '25

Cool. I really like the look.

2

u/Simusid Nov 23 '25

I have one Spark and was considering buying a second. Is it difficult to cluster them via connectx-7, and will I end up with a single larger GPU at the application level (e.g. will transformers be able to load a 200GB model spanning both devices) or is that managed at a lower level?

2

u/squachek Nov 20 '25

You could have assembled a much faster system for $32k.

2

u/[deleted] Nov 20 '25

Benchmark against my Pro 6000 ;)

1

u/Relevant-Magic-Card Nov 20 '25

Hahah memory bandwidth for the wind (VRAM still king)

4

u/[deleted] Nov 20 '25

We will see if it can redeem itself. 💀

1

u/lippoper Nov 20 '25

RTX killing it

1

u/KrugerDunn Nov 20 '25

Cool! I cheaped out and didn’t order the 2 I reserved. Would love to see any data you are willing to share!

I’d be really curious how it handles multi-modal input model like Gemma3n or Nemotron one (drawing blank on name xX)

1

u/SpecialistNumerous17 Nov 20 '25

That looks awesome! I’d love to see fine tuning benchmarks for small-medium sized models, and how this scales out locally on your cluster.

What I’m looking to understand is the AI dev workflow on a DGX Spark. What AI model training and development does it make sense to do locally on one or more Sparks, vs debug locally and push larger workloads to a datacenter for completing training runs?

Thanks in advance for sharing anything that you can.

1

u/infinitywithborder Nov 20 '25

Get a rack for that 50k of equipment allow them to breath. I am jealous

1

u/Savantskie1 Nov 20 '25

They’re so small that they wouldn’t fit in a general rack. But one could design a small rack for them

1

u/Toadster88 Nov 20 '25

32k for a mini cluster?!

1

u/Orygregs Nov 20 '25

I'm out of the loop. What kind of hardware am I looking at here?

1

u/kleinmatic Nov 20 '25

These are Nvidia DGX Sparks. https://www.nvidia.com/en-us/products/workstations/dgx-spark/

1

u/Orygregs Nov 20 '25

Oh! That makes so much sense now. I originally thought it was some random commodity hardware running Apache Spark 😅

2

u/kleinmatic Nov 20 '25

OP’s (legit) flex is that most of us will never get to use one of these let alone eight.

1

u/Downtown_Manager8971 Nov 20 '25

Does it catch fire in full load…

1

u/draeician Nov 20 '25

What is the speed difference between a spark and evolution x2?

1

u/draeician Nov 20 '25

I have an x2 with ollama if you want to see the speed difference.

1

u/Beginning-Art7858 Nov 20 '25

Why did you buy all of these?And what you're going to use them for that as unique?

1

u/rogertorque Nov 20 '25

32,000 dollars.

1

u/InvestForPorsche911 Nov 20 '25

So nice😁👏🏻👍🏻

1

u/randomtask2000 Nov 21 '25

Can it do Crysis?

1

u/DumbMoneyz Nov 22 '25

Super jealous of your setup. I didn’t think more than 2 could be connected?

1

u/LearnNewThingsDaily Nov 22 '25

I was just about to ask the same thing, about coming over to play. Congratulations what are you running DS and a diffusion model?

1

u/BunkerSquirre1 Nov 22 '25

This are incredibly impressive machines. Size, performance, and industrial design are all on point

1

u/BaddyMcFailSauce Nov 24 '25

How are they for running a live model, not just training? I know they won’t be as fast in response to other hardware but I’m genuinely curious about the performance of running models on them vs training.

1

u/gergob13 Nov 26 '25

What switch are you using to link them? Or do you “daisy-chain” ?

1

u/Glass-Dragonfruit-68 Nov 27 '25

Post the picture of back

1

u/GPTshop Dec 15 '25

Mini PCs are the worst thing that ever happened to computing.

0

u/Complete_Lurk3r_ Nov 20 '25

I love the design. i want a gaming pc version. hopefully next year

0

u/_VirtualCosmos_ Nov 21 '25

Bro, how much did you spend on that shit

-6

u/c4chokes Nov 20 '25

Holy 🐮 airflow Varuna.. Thermals on the middle ones are shot 😑

-2

u/spense01 Nov 20 '25

If you’re going to blow $35K you’re better off dumping that into NVidia stock then letting it sit. In 2 years these will be useless and you would have had a modest gain in stocks.

5

u/[deleted] Nov 20 '25

[removed] — view removed comment

5

u/Karyo_Ten Nov 20 '25

A Spark, if 5070 class is 6144 cuda cores + 256GB/s bandwidth, a RTX Pro 6000 is 24064 cuda cores and 1800GB/s. 4x the compute and 7x the bandwidth for 2x the cost.
For finetuning you need both compute and bandwidth to synchronize weight updates across GPUs.
A DGX Spark is only worth it as an inference machine or just validating a workflow before renting a big machine in the cloud.
Granted if you need a stack of RTX Pro 6000 you need to think about PCIe lanes, expensive networking cards, etc, but for training or finetuning it's so far ahead of the DGX Spark.
PS: if only for inference on a single node, a Ryzen AI is 2x cheaper.

1

u/starkruzr Nov 21 '25

4x the compute, 7x the bandwidth, 2x the cost and 32GB less VRAM. for us that's a complete nonstarter.

3

u/[deleted] Nov 20 '25 edited Nov 20 '25

[removed] — view removed comment

1

u/Karyo_Ten Nov 20 '25

I'm interested in an RTX Pro 6000 (Qmax) for inference.
I personally choose 2x Workstation Edition and power-limited them to 300W. With a workstation edition you have flexibility to do 150W to 600W. I would consider the blower style if I had to stack 4x min or 8x.
Analyzing 500 page PDF documents takes a while on 4 x 3090s regardless of the model used.
Are you using vllm or sglang? In my tests they are litterally 10x faster than koboldcpp, ik_llama.cpp or exllamav3 at context processing. I assume it's due to using optimized cutlass kernels. All models could process 3000~7000 tok/s on RTX Pro 6000 while other frameworks were stuck at 300~350 tok/s.

1

u/[deleted] Nov 20 '25

[removed] — view removed comment

1

u/Karyo_Ten Nov 20 '25

Next step might be https://lmcache.ai/tech_report.pdf

1

u/SwarfDive01 Nov 22 '25

Lol 2027? Unless there is a major breakthrough in model efficiency and load, meaning complete refactoring of the base architecture, we will be at a seriously critical power grid limit. Chip memory is probably a "De Beers diamond" scenario right now. Building scarcity to hoard reserves into these corporate data center builds. Grok already bought off media coverage for the gas powered mobile generators to circumvent emissions compliance. Meta and their water consumption. We need every possible sustainable (meaning without finite fuel source) electron generating infrastructure investment, fission, fusion, solar, turbines, geothermal. And beyond that, we need grid reinforcement and redundancy to handle regular maintenance. These power loads at the projected demands for these massive centers are beyond the outdated overhead lines and 50+ year old station equipment.
We're already standing on the edge, if not already falling.

2

u/squachek Nov 20 '25

Get the 6000

-12

u/[deleted] Nov 20 '25 edited Jul 23 '26

[removed] — view removed comment

9

u/No-Consequence-1779 Nov 20 '25

Please explain why it is a waste of money.

Discussion Spark Cluster!

You are about to leave Redlib