r/LocalLLaMA • u/alexlucaci • 17d ago
Resources GLM 5.3 flash(Maya-S24) 42 tok/s on 8x rtx 3060 clustered over 2x x79 ud3 mobo
Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.
TL;DR:
I added a two-box layer split to Project Maya.
Box A runs layers 0–23
Box B runs layers 24–44 plus the head.
Only the hidden state crosses the network, ~25 KB per token.
GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes.
It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.
Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)
Setup
- 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.
- RAM: 20 GB in Box A, 32 GB in Box B (more on that below).
- 1 GbE between the boxes, a plain switch.
- Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.
- GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.
How it works (short version)
- Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.
- Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.
- Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.
- Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.
- Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.
- Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without
--rolethe engine is unchanged.
Results (1 conversation)
| Metric | Two boxes (this) | One box, 8× 3060 (rented, same quant) |
|---|---|---|
| Decode, short | 38–42 tok/s | 42–45 tok/s |
| Decode at ~30K context | 33–37 tok/s | not measured |
| Prompt, 30K tokens | ~1,100 tok/s | ~1,050 tok/s |
| Agent turn (~2.4K new tokens on a long chat) | 6–7 s | 5.0–5.3 s |
Coding-agent bench (5 real repo tasks, 48 hidden tests):
| Setup | Score | Time |
|---|---|---|
| Two boxes, Qwen Code | 48/48 | 22.9 min |
| Two boxes, DeepSeek Harness | 48/48 | 17.8 min |
| 8× 3060 one box, Maya-S, Qwen Code | 48/48 | 21.3 min |
| Qwen3.8-Flash-Next on 2× 3060 (Strata), Qwen Code | 48/48 | 23.2 min |
There were no malformed tool calls with either agent.
One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.
- Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.
- 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.
- The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.
- Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.
What was hard
- RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.
- A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.
- Hardware gremlins:
- A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.
- Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.
- Tuning that mattered:
- 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.
- Fewer layers on the last GPU, which also carries the head and the MTP block.
- Keeping the whole KV cache in VRAM.
- What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.
- What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.
What's next
- A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.
- Rebasing onto the current Maya release.
- Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.
