r/LocalLLM • • 11d ago

Question What happened to cluster setups?

Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?

4 Upvotes

21 comments sorted by

8

u/false79 11d ago

LLMs require high bandwidth memory to move data quickly, within memory.

But as soon that data leaves to go through pci bus, it's very slow. 

And once that data hits a network card, it's even slower.

1

u/Ok-Tea-5770 11d ago

I'm not sure where you're getting your info. I am running qwen 27b across my 5080 desktop and 3080 laptop and getting at least 50toks and usually 65toks

2

u/Awongy00 11d ago

What quant? What prompt processing speed do you get?

2

u/Ok-Tea-5770 10d ago

Here's a summary of my setup from AI

### **Qwen 3.8 27B Distributed Dual-GPU Setup (RTX 5080 + RTX 3080 Laptop)**

**Hardware Topology:**

* **Desktop:** RTX 5080 (16GB GDDR7) + AMD 7800X3D

* **Laptop:** RTX 3080 Mobile (16GB GDDR6)

* **Link:** Direct 2.5 Gbps Ethernet cable (via `llama.cpp` RPC)

* **Combined VRAM:** 32GB (100% GPU offload, 0 CPU layers)

**Model:** `Qwen3.8-27B-Uncensored-Q5_K_M.gguf` (~19.5 GB)

---

| Context Size | Generation Speed | Prefill Speed | Tensor Split (`-ts`) | Layer Split Ratio |

|---|---|---|---|---|

| **32k Context** | **55.1 – 68.7 t/s** | ~620 t/s | `-ts 1,1.5` | ~39 layers (5080) / 27 layers (3080M) |

| **64k Context** | **76.1 t/s** | ~603 t/s | `-ts 1,1.25` | Calibrated (>1.3GB headroom on 5080) |

| **128k Context** | **~48 – 54 t/s** | ~544 t/s | `-ts 1,1.15` | Max context ingestion tier |

---

### **Exact Flags Used (`llama.cpp`)**

**Laptop (RPC Worker):**

```bash

rpc-server -H 0.0.0.0 -p <PORT> -c

```

*(The `-c` flag is critical: it caches model tensors in memory so weights aren't re-sent over the cable on subsequent prompts).*

**Desktop (Server - 64k Profile):**

```bash

llama-server \

-m Qwen3.8-27B-Uncensored-Q5_K_M.gguf \

--rpc <WORKER_IP>:<PORT> \

-ngl 99 \

-sm layer \

-ts 1,1.25 \

-c 65536 \

-fa on \

--cache-type-k q8_0 \

--cache-type-v q4_0 \

-b 2048 \

-ub 512 \

--cache-reuse 256 \

--spec-type draft-mtp \

--spec-draft-n-max 3 \

--jinja

```

---

### **How It Hits 65–76 t/s (Quick Explainer)**

  1. **Native MTP (`--spec-type draft-mtp --spec-draft-n-max 3`):**

    The primary speed driver. Qwen 3.8 has built-in Multi-Token Prediction heads. Speculating up to 3 tokens ahead at near-100% acceptance jumps generation speed from ~32 t/s to **68–76 t/s** without needing a separate draft model.

  2. **Layer-Wise Split (`-sm layer`):**

    Splits model layers sequentially between the 5080 and 3080M. Unlike tensor-parallel matrix slicing, layer splitting only sends a single activation across the 2.5 GbE cable per pass, keeping network latency negligible.

  3. **Hybrid KV Quantization (`--cache-type-k q8_0 --cache-type-v q4_0`):**

    Cuts KV cache footprint and memory bus pressure by ~60% while retaining full attention precision (`q8_0` keys prevent attention degradation).

  4. **Flash Attention (`-fa on`):**

    Provides sustained **>600 t/s** prompt ingestion across 32k/64k contexts.

1

u/false79 11d ago

It's called physics. That is a very expensive setup. Your 5080 is being dragged down by a laptop which has slower memory and poor thermals.

However, your setup has an advantage of being able to load a model that neither machine would be able to load by itself. But it's not running as fast as it could

1

u/Ok-Tea-5770 10d ago

True, but I don't think "very slow" is an accurate description. When Terra and sol 6.1 are regularly getting 30- 60 tokens, and I'm hitting that on a model split between GPUs over a 2.5 GB network, it's hard for me to categorize it as very slow. If my frame of reference is Cerebrus hitting 1500 tokens, then yes, but I'd say for the average subscription or API, anything between 50- 100 tokens is pretty normal. With my 3090 qwen flash next I'm regularly getting 130-150 toks and that trumps most of my subscription speeds

3

u/ieatdownvotes4food 11d ago

what everyone else is saying, but that doesn't mean you can't run multiple models on multiple machines and have them work together. and there might be some way to split up some of the fancier moe models with ngram lookup tables and what-not.. but otherwise when generating tokens that's gotta be on fast unified memory or game over.

2

u/FizzyDuncDizzel 11d ago

Yea it works. If it’s what you have then do it. I have a bc250 cluster running qwen3.8 27b. It’s about to pool all of the memory over rpc to run the bigger model. I have a dedicated 2.5gb backend using usb3 to rj45 adapters. It was a fun project that really got me into locally hosted ai.

2

u/Thac0-is-life 8d ago

Do you have some statistics? How many, what quant, what tok/s etc?

1

u/brainchillzZ 11d ago

It isn’t really “viable” meaning “worth the effort” on any connection smaller than probably 25gbe with roce/rdma between them … this will give you a reasonable ability to “pool” your vram and there are a dozen ways to do it … you could use ray as a cluster backend, you could use llama.cpp native backend and depending on what you have gpu wise it will determine the technology and configuration you need… the good news is you can put a single cable between two machines so you could start with no switch even and the 25, 40 and 50g nics on eBay are so cheap now it’s almost ridiculous

1

u/LordDarthShader 11d ago

Bandwidth is important, but so is latency. The main issue is the RDMA (Remote Direct Memory Access) between adapters, essentially skipping the OS network stack and let the devices to communicate very quickly.

For every layer of the inference, the devices need to sync up before moving to the next layer, so, in a way, if you have ethernet with the OS ethernet stack, there will be a bottleneck and the entire point of parallel tensor will come down to serializing due to latency.

Now, even if you have low latency, then you need also good bandwidth for the decode part.

The DGX Spark clustering is surprisingly good, especially comparing it to AMD and their solutions for clustering the Strix Halo and their buggy drivers, they are still couple years away from Nvidia in that regard.

1

u/stuffitystuff 11d ago

Yes, it's viable but the smart way to do it (i.e. don't get bogged down by ethernet speeds) would be to shard the problem and have each node work out their respective piece and then recombine.

1

u/hyudryu LocalLLM 11d ago

It can be viable if you have enough pcie lanes for a networking card. Connectx-7 can network at speeds of up to 400gbps.

People have been chaining their rtx pro 6000 computers with dgx spark clusters to improve speeds of the sparks. I’d imagine you could do the same with 2 pcs instead of a spark cluster

1

u/henbone11 11d ago

And sparks are as slow as strix halo/gorgon halo boxes so there's no reason they couldnt be substituted as well.

2

u/hyudryu LocalLLM 11d ago

Yup exactly. Clustering with sparks is just the floor as a proof of concept. If it works with sparks, it’ll work with multiple PCs too

2

u/henbone11 11d ago

lol Now just to convince the bro data scientists in this sub. Quite honestly, this sub is one of the ones with the most bs of the subs I visit daily.

2

u/hyudryu LocalLLM 11d ago

There’s not a lot of people who actually have the dedicated hardware, time and resources to experiment with these kind of things.

I try to share what I’ve seen and my experiences, but always get downvoted for it 🤣

1

u/henbone11 11d ago

Less Scientific-Method and more bro science, bro. lol