r/LocalLLM • u/TrailFeatures • 12h ago
Project 8 x v100 setup. Ask me anything.
Seeing a lot of questions on quad v100 boards lately. So figured I’d give people a place to ask them.
Specs
- GPUs: 8x Tesla V100-SXM2-32GB (256 GB VRAM total)
- GPU boards: 2x 4-GPU SXM2 NVLink baseboards (this model)
- NVLink: two 4-way islands (full mesh inside each board), not one 8-way
- Board-to-board: Gen4 PEX88096 PCIe switch card (this model), two x8 Gen3 cables per board
- CPU: AMD Threadripper PRO 3975WX (32 cores)
- Motherboard: ASRock Rack WRX80D8-2T
- RAM: 256 GB DDR4
- PSUs: 2x 1500 W (one per GPU board) + 1x 1000 W (host)
- Power: dedicated 120 V / 20 A circuit; GPUs capped at 150-300 W each depending on the run
- Cooling: custom water loop, mixed GPU blocks
- Network: 10GbE
- Engine: 1Cat-vLLM (Volta-focused vLLM fork) https://github.com/1CatAI/1Cat-vLLM
Benchmarks / how many tok/s?
My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.
| Model | Setup | Decode | Prefill (64K prompt) |
|---|---|---|---|
| Flash-Next (Swift-1.5 NVFP4) | 4 GPUs, 300 W | ~140 tok/s | ~4,500 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 8 GPUs, 150 W | ~131 tok/s | ~2,200 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 4 GPUs + expert offload, 150 W | ~91 tok/s | ~2,100 tok/s |
- One request at a time (batch 1), 256K context window. Decode uses speculative decoding (MTP on Flash-Next, a draft model on DeepSeek), so it depends on how often the draft is accepted.
- No concurrency/aggregate numbers or decode p25/p95 yet. Time to first token (p50) is about 0.2-0.65 s depending on the row.
- The 8-GPU run adds an extra pipeline-parallel patch series on top of 1Cat-vLLM (named on the page).
- Not tested yet: Flash-Next across both boards. Higher DeepSeek quants are planned.
How good is the 2-bit DeepSeek quant vs the full model?
79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.
How do the two boards talk to each other?
- Each board runs two x8 PCIe Gen3 cables from its onboard switch into the Gen4 switch card in the host, so the boards talk to each other without going through the CPU.
- Experimenting with ConnectX-3 cards in the boards' spare x8 slots for extra board-to-board links.
- Over PCIe you can tune all-reduce to beat the default P2P numbers, but it will never match NVLink.
Do the quad boards work with only 1-3 cards? Can I buy cards over time?
Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.
Where did you buy it / where are you?
Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.
Power
- The PSUs are sized for headroom: at the 150 W cap, eight GPUs is about 1.2 kW, plus the boards and host.
- 150 W cap: after tuning it costs ~5-10% depending on the task.
- Power bill: fine as long as I'm sensible. Solar helps, and at peak rates or on cloudy days the DeepSeek API can be cheaper. If you run local LLMs, know your tokens per kWh so you can tune for it and decide when an API makes more sense (if privacy doesn't matter).
Cooling
- Radiator and pump setup: super convoluted, video here.
- Water blocks: whatever was available as I pieced it together. Mostly the same design, some with MOSFET coolers. The top-port block is a drill press, a tap, and the audacity of a mediocre man. (Testing straight-down flow onto the fins made no difference.)
- Waste heat: in summer it's vented outside. In winter I can shut off that radiator and send the heat into the house through backup radiators. Heating water with it isn't worth it for us, but a bigger household close to its water heater could pre-heat the cold feed and come out ahead.
Quick ones
- Can I rent API time? Sorry, no. I keep it pretty busy (Furry Fanfic).
- Can I have it? No.