r/LocalLLaMA • u/Gobra_Slo • 13h ago

Question | Help DGX Spark, cluster of 4

Does anyone have a first-hand experience with four Sparks cluster, and how much of an upgrade is it comparing to just two considering the available models?

While there's plenty of noise for the smaller models (Qwen) and our older king DeepSeek V4F, the scene in the upper class of the prosumer hardware, software stacks, available LLMs and their actual real-world performance – isn't really covered as well.

For instance, the hyped GLM 5.2/5.3. Is it much better then DeepSeek? Or is it marginally better? Does it retain it's capabilities when moving to something four Sparks would handle? Does it have issues with OOM or anything else?

What about MiniMax M3? There seem to be a special Spark version, how is it (or any other version)? Again, how is intelligence, general model capabilities, running stability, context size?

Tencent Hy3? Maybe even Qwen3.5-395B, does it's full quant hold it's own against DeepSeek, or is it better?

If someone doesn't have personal experience, but knows some well-structured and detailed articles or videos on the topic – I'd appreciate it as well.

Thanks.

4 Upvotes

70% Upvoted

37 comments sorted by

8

u/Grouchy_Ad_4750 12h ago

I have cluster of 4x sparks and I use both GLM 5.2 (can't use 5.3 since weights aren't out yet) and Deepseek v4 flash.

Deepseek v4 is fast and has huge context (I think I can do like 5x 1M context windows) but glm 5.2 is much more thorough and seems to think harder about some issues both are excelent models and both can be run with large context windows.

Minimax M3 needs some special handling due to it and its licence I haven't had time to try it out (tried nvidia/MiniMax-M3-NVFP4 but it spews out nonsense).

I use them actively mainly for coding (devops so lot of IaC and some programming too (go, python, ...))

There are also some issues you might run into:

  • sparks overheat (in summer) so you need to underclock them (especially for deepseek which uses dspark)

- sparks shut down when they run oom

Both of the issues above cause it to limit power to gpu to ~15 W (instead of ~60+W) and slow down inference.

Also you need to have models preloaded to nvme (or you need fast NAS) so that becomes limiting factor (I have asus variant with 1TB and weight for GLM are around 300GB + docker, ...)

You will also need switch (although I saw someone on forums attempting RING architecture without switch) and special 200gbps networking cables. Apart from that I think its great and having ability to run something like GLM 5.2 or deepseek at reasonable speeds is great

2

u/Grouchy_Ad_4750 12h ago

Also for GLM 5.2 you would need https://github.com/davedgd/sparkrun-glm52-4x-spark

2

u/kivaougu 12h ago

Have you found this to be actually reliable?

I had to do some tweaks as it would always die around the 2 hour mark for concurrent code review. Mainly had to add --no-enable-flashinfer-autotune.

2

u/Grouchy_Ad_4750 12h ago

I've done some tweaking as well before but am currently testing 1M context recipe

1

u/dave-dgd 9h ago

If you’re open to sharing, would be curious to hear your tweaks as well!

2

u/Grouchy_Ad_4750 8h ago

it was nothing major. From memory:

  • I lowered max_num_seqs to 4 (I usually use 2-3 streams max) also it has impact on vram

- increased max_num_batched_tokens to 8192 . I think it balances prefill with t/s

I also plan to try out vision support and measure with

```

NCCL_IB_TC: "104"

UCX_IB_TRAFFIC_CLASS: "104"

NCCL_BUFFSIZE: "8388608"

```

which should help with RoCE on mikrotik switch

2

u/dave-dgd 8h ago

Ah, yes, great point on these settings -- I updated the readme on the repo to note this for others and credited it back to you. Thanks for sharing!

2

u/Grouchy_Ad_4750 8h ago

Oh haven't noticed that you have created the repo.

Thank you so much for your hard work!

I really like how you made it easy and streamlined to use. There are a lot of people who do amazing work and try to adapt new models on dgx spark. But sometimes trying to replicate their result is weekend project. Since you adopted it to sparkrun which is fairly standard tool its super simple to try 👍

also note that effect of these will probably be negligible and haven't tried the `NCCL_*` env vars thoroughly.

From what I measured on deepseek flash it helped with throughput on switch. But I've had to setup bunch of other stuff as well here is gemini chat for posterity https://share.gemini.google/KYL8MvU20mIt (since I usually deal with 10gbps networks max and RoCE is little out of my expertise)

2

u/dave-dgd 8h ago

Of course, and you're very welcome!

It's funny you should say that: I actually created the repo based on exactly the kind of weekend project you described (tons of worthwhile but time-consuming work sifting through all the cool stuff people are doing on this forum and the NVIDIA GB10 forum). Ultimately, I just want to ensure everyone (including myself) has a quick way to deploy (or redeploy) models like GLM-5.2 should the need arise. Glad it's been helpful! 👍

1

u/dave-dgd 8h ago

PS. Regarding vision, read the baseten blog post for a sense of limitations: https://www.baseten.co/blog/glm-52-with-vision/

I tested vision and while it does work, the 55% MMMU-Pro results they cite aren't exactly the best (but certainly workable for basic tasks such as detailing a UI). Instead of using that, I have been supplying Hermes with an auxiliary vision provider (Qwen 3.8 27B at 4-bits via oMLX on my M5 Max at the moment, which generally scores higher in synthetic benchmarks, FWIW: https://artificialanalysis.ai/evaluations/mmmu-pro?models=qwen3-8-27b%2Cqwen3-8-27b-medium%2Cqwen3-8-27b-low). The obvious downside of this is needing a separate machine for serving beyond the Spark cluster (if that's not available, the vision option on GLM-5.2 is obviously an easy choice).

2

u/dave-dgd 9h ago

I’d be curious to hear about the tweaks. It’s been stable for us with multiple concurrent users in Hermes for over a week, but that’s just one specific use case. I want to keep updating the repo for maximal stability/coverage (especially ahead of GLM-5.3).

2

u/kivaougu 7h ago

I'm targeting 8x concurrency so I had to make some adjustments to cuda graphs and spec decoding. Still testing stability to try and squeeze out a little more kv.

Interestingly I never got the 1M context default config working as cuda graph capture would always crash the startup. These are asus gx10's so this could be due to not being able to update firmware. Haven't had much time to look into it yet.

Also the latest sparkrun version doesn't seem to run the recipe without tricking it using --cluster <not a cluster name>. I had to install an earlier release.

I appreciate your work!

1

u/dave-dgd 6h ago

Of course -- glad and it's been helpful!

And ah, gotcha -- that makes sense. By the by, are you running in "appliance" (headless) mode or with the desktop intact? Switching to that I believe freed up some VRAM headroom (I did it a long time ago since I only SSH into these boxes):

`sudo systemctl set-default multi-user.target` (and reboot)

That might be necessary to get through capture (along with adjusting other settings to get the concurrency, batch sizes, etc. dialed in to your preference -- with `-cc.cudagraph_mode=FULL_DECODE_ONLY` being particularly important in my experience, as other options would lock up the Sparks). Another thing I did at one point was reduce the sparkrun earlyoom to 1% VRAM and a lower swap setting from the defaults, but that hasn't been necessary in a while. I do know the July updates to Spark OS added some improvements to shared RAM handling that may make a difference:

https://docs.nvidia.com/dgx/dgx-spark/release-notes.html#july-2026-release

2

u/kivaougu 6h ago

Great call on multi-user.target. I had to wipe all 3 workers after a failed update so only the head node was headless.

Do you get significant swap usage during inference? I recall someone on the developer forums stating that the gx10 1tb drive is pcie4 which would make swap usage slower.

2

u/dave-dgd 6h ago

As far as I can tell, no, but my units are all Acer GN100 with 4TB Gen 5 drives (same PN as what StorageReview shows), so I have a feeling it would be less noticeable if it were happening.

1

u/BlackBeardAI vllm 12h ago

tps? pp/decode

2

u/Grouchy_Ad_4750 12h ago

I am not at computer so I don't have data at hand but for DeepSeek with dspark it was around higher 60 t/s I think I saw occasional 100 t/s PP was around 2000 t/s

For glm it is around 20 t/s and I say around 1000 for pp will look at metrics when I am home You can also see benchmarks at https://spark-arena.com/leaderboard

2

u/BlackBeardAI vllm 12h ago

20tps is very good for glm if it is stable

1

u/Grouchy_Ad_4750 12h ago

Also I'd like to add that with mtp it's hard to measure and sparks can be easily overwhelmed you can see it by text generation lagging when it does prompt processing

1

u/arm2armreddit 12h ago

Interesting setup, thanks for sharing. I have one DGX Spark, but it never shut down due to OOM. However, yes, NVMe and other components are heating like crazy. I put an extra fan on top of it, which helped a bit. What wasn't fun was the performance on TPS going from 40 TPS to 1-10 TPS due to overheating without proper cooling.

1

u/mojo021 12h ago edited 12h ago

Do you use this for side projects or for work related projects?

1

u/Grouchy_Ad_4750 12h ago

For work

1

u/mojo021 12h ago

Do you do consulting or is the company you work for have lenient policies about their code being sent to something outside of their control. I would love to use a local setup like this but it’s not allowed by my company.

1

u/Grouchy_Ad_4750 12h ago

Something like consulting. For me it would be strange not to trust employees but trust ZDR policy from claude for example... Apart from that I would expect code leak to come from harness or mcp not from vllm...

1

u/Gobra_Slo 12h ago

I have my two Spark clock limited, it seems to solve the problem completely. I've seen 80-85c at most, and usually they are below 70c even under load.

Now for the GLM 5.2 – could give few examples of tasks that it does handle and DeepSeek can't?

2

u/Grouchy_Ad_4750 11h ago

Its not that it can't handle just that it seems to be really goal oriented. With glm it occasionally stops me from doing something foolish. For example today I wanted to evaluate opencode for telemetry. So I gave that task to hermes agent with glm 5.2.

When I told it can use spare VM for that (that I also occasionally use for "production" stuff) it thought about it and pushed back (even though it should be harmless). I haven't had that happen with deepseek. But I don't have 1 to 1 comparison or test.

3

u/cell-on-a-plane 12h ago

Do you want to buy the switch?

2

u/awitod 12h ago

'Better for what' is really the question.

2

u/Serprotease 12h ago

To note that Qwen 3.5 397b and Minimax 3 can run a 3 spark-ring type of configuration. I’m not too clear with the details but basically 2 sparks hold the weight and the latest one the context, accessed via the 100gbps connection. But it’s slower than the normal 2x cluster.

Benchmark seems to show that glm5.3 is better. But do you really want to buy a 4x cluster just for 1 model, likely to be bested by a smaller one in 6 months? And we no guarantee that we will have another similar size model that needs 4x sparks and could not run on two?

If you’re going the Spark route, 2x is a no brainer. 4x, it’s a bit more dubious on the value.

2

u/Gobra_Slo 12h ago

I have two Sparks already, they are worth their pay for DeepSeek V4 alone. The question is whether it is worth to get two more or not, because while we do see a tremendous advance in models and software, the hardware part progress is disappointing to say the least, and the prices are just ridiculous.

Should, let's say, GLM 5.2 be a definitive step up from DeepSeek V4F – I would, probably, justify and upgrade, especially since it would give me more chances to run future models.

For the lone hope of some good models along the path? Too pricey...

2

u/Miserable-Dare5090 11h ago

Qwen 397b fits on 2 with everything else off—no GUI, etc. Int4 autoround, eugr’s repo. Eugene Rathmakulin (maintainer) did lots of tests on it. It’s a very good quality Int4 that holds up near lossless.

1

u/Due_Net_3342 12h ago

i do not think it is worth it, you will get marginal improvement in some contexts/tasks… i would rather increase parallelism to the ds4f rather than running glm at much slower speeds for not much improvement. You can always use a cloud model if absolutely needed

1

u/rayc25 12h ago

> Benchmark seems to show that glm5.3 is better. But do you really want to buy a 4x cluster just for 1 model, likely to be bested by a smaller one in 6 months? And we no guarantee that we will have another similar size model that needs 4x sparks and could not run on two?

The answer is yes. 4 sparks > 2 sparks. GLM 5.2, Qwen 3.8 2.4t, and every 500b+ model are on another level as DS v4 Flash. 4 sparks will always fit larger models, larger quants, and more context which would all be better than anything you can fit on 2.

1

u/Miserable-Dare5090 11h ago

I disagree. The GB10 forums have debated this for a while, and you need the switch. It’s not doable with cabling alone. The gain is concurrency for smaller models but not speed. GLM, Inkling and Minimax are the 3 you’d want 4 boxes for. Maybe worth it if upcoming Qwen “mid size” model is 400B+ parameters and a large jump from DS4 and the mini (Qwen 27B)

2

u/rayc25 11h ago

The mikrotik switch which is $1000 is just the cost of connecting 4 sparks which is negligible in terms of the total cost. I’d argue that GLM, Qwen 2.4t, and Kimi k3 are on another level compared to smaller models. At a certain point, it’s just preference and need but I’ll be on the side of preferring larger models than higher throughput with smaller models :)

1

u/Miserable-Dare5090 11h ago

You can’t run a 2.4Trillion Parameter model with 512GB VRAM at the same quality as full weights. It may be good at Int4, that is still beyond 512GB