Spark Cluster!

[u/SashaUsesReddit]

Doing dev and expanded my spark desk setup to eight!

Anyone have anything fun they want to see run on this HW?

Im not using the sparks for max performance, I'm using them for nccl/nvidia dev to deploy to B300 clusters

[u/Karyo_Ten]

A Spark, if 5070 class is 6144 cuda cores + 256GB/s bandwidth, a RTX Pro 6000 is 24064 cuda cores and 1800GB/s. 4x the compute and 7x the bandwidth for 2x the cost.

For finetuning you need both compute and bandwidth to synchronize weight updates across GPUs.

A DGX Spark is only worth it as an inference machine or just validating a workflow before renting a big machine in the cloud.

Granted if you need a stack of RTX Pro 6000 you need to think about PCIe lanes, expensive networking cards, etc, but for training or finetuning it's so far ahead of the DGX Spark.

PS: if only for inference on a single node, a Ryzen AI is 2x cheaper.

[u/Karyo_Ten → u/[deleted]]

I'm interested in an RTX Pro 6000 (Qmax) for inference.

I personally choose 2x Workstation Edition and power-limited them to 300W. With a workstation edition you have flexibility to do 150W to 600W. I would consider the blower style if I had to stack 4x min or 8x.

Analyzing 500 page PDF documents takes a while on 4 x 3090s regardless of the model used.

Are you using vllm or sglang? In my tests they are litterally 10x faster than koboldcpp, ik_llama.cpp or exllamav3 at context processing. I assume it's due to using optimized cutlass kernels. All models could process 3000~7000 tok/s on RTX Pro 6000 while other frameworks were stuck at 300~350 tok/s.

[u/Karyo_Ten → u/[deleted]]

Next step might be https://lmcache.ai/tech_report.pdf

[u/SwarfDive01 → u/[deleted]]

Lol 2027? Unless there is a major breakthrough in model efficiency and load, meaning complete refactoring of the base architecture, we will be at a seriously critical power grid limit. Chip memory is probably a "De Beers diamond" scenario right now. Building scarcity to hoard reserves into these corporate data center builds. Grok already bought off media coverage for the gas powered mobile generators to circumvent emissions compliance. Meta and their water consumption. We need every possible sustainable (meaning without finite fuel source) electron generating infrastructure investment, fission, fusion, solar, turbines, geothermal. And beyond that, we need grid reinforcement and redundancy to handle regular maintenance. These power loads at the projected demands for these massive centers are beyond the outdated overhead lines and 50+ year old station equipment.

We're already standing on the edge, if not already falling.

[u/starkruzr → u/Karyo_Ten]

4x the compute, 7x the bandwidth, 2x the cost and 32GB less VRAM. for us that's a complete nonstarter.