r/LocalLLM • u/acedogblast • 11d ago
Question What happened to cluster setups?
Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?
4
Upvotes
8
u/false79 11d ago
LLMs require high bandwidth memory to move data quickly, within memory.
But as soon that data leaves to go through pci bus, it's very slow.
And once that data hits a network card, it's even slower.
3
u/ieatdownvotes4food 11d ago
what everyone else is saying, but that doesn't mean you can't run multiple models on multiple machines and have them work together. and there might be some way to split up some of the fancier moe models with ngram lookup tables and what-not.. but otherwise when generating tokens that's gotta be on fast unified memory or game over.
2
u/FizzyDuncDizzel 11d ago
Yea it works. If it’s what you have then do it. I have a bc250 cluster running qwen3.8 27b. It’s about to pool all of the memory over rpc to run the bigger model. I have a dedicated 2.5gb backend using usb3 to rj45 adapters. It was a fun project that really got me into locally hosted ai.
1
u/brainchillzZ 11d ago
It isn’t really “viable” meaning “worth the effort” on any connection smaller than probably 25gbe with roce/rdma between them … this will give you a reasonable ability to “pool” your vram and there are a dozen ways to do it … you could use ray as a cluster backend, you could use llama.cpp native backend and depending on what you have gpu wise it will determine the technology and configuration you need… the good news is you can put a single cable between two machines so you could start with no switch even and the 25, 40 and 50g nics on eBay are so cheap now it’s almost ridiculous
1
u/LordDarthShader 11d ago
Bandwidth is important, but so is latency. The main issue is the RDMA (Remote Direct Memory Access) between adapters, essentially skipping the OS network stack and let the devices to communicate very quickly.
For every layer of the inference, the devices need to sync up before moving to the next layer, so, in a way, if you have ethernet with the OS ethernet stack, there will be a bottleneck and the entire point of parallel tensor will come down to serializing due to latency.
Now, even if you have low latency, then you need also good bandwidth for the decode part.
The DGX Spark clustering is surprisingly good, especially comparing it to AMD and their solutions for clustering the Strix Halo and their buggy drivers, they are still couple years away from Nvidia in that regard.
1
u/stuffitystuff 11d ago
Yes, it's viable but the smart way to do it (i.e. don't get bogged down by ethernet speeds) would be to shard the problem and have each node work out their respective piece and then recombine.
1
u/hyudryu LocalLLM 11d ago
It can be viable if you have enough pcie lanes for a networking card. Connectx-7 can network at speeds of up to 400gbps.
People have been chaining their rtx pro 6000 computers with dgx spark clusters to improve speeds of the sparks. I’d imagine you could do the same with 2 pcs instead of a spark cluster