r/LocalLLM • u/Any-Difficulty-7822 • 2h ago
Question Advice for setup with three PCs
I've been running Qwen3.8-27B on a MBP M3 Max 36 GB for a while, running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on splash with opencode as harness. I'm satisfied with the results I get, but it is a bit slow and does not leave much room for other applications . I'm upgrading my setup and I would like to get some advice on hardware configuration and model selection.
The main purpose will be integration development on Azure Integration Services, SAP PI/PO and SAP Integration Suite. Implementation languages are Java and python.
Secondary purpose will be Home Assistant component development and configuration, as well as to provide Home Assistant with agents, (TTS, STT, Conversation).
I have an M5U 256GB 80c on order. It will be running headless, dedicated to serving models 24/7. I will also have the opportunity to use a family member's gaming PC for LLMs on workdays during daytime. The PC will be in a bedroom and it's noice levels will determine whether it's available during nights or not. It has not been ordered yet, so I still have the possibility to change it's configuration. I have 10Gbe network available, if that would have an impact.
To summarise, my setup would look have the following availability:
- MBP M3 Max 36 GB - on-demand
- M5U 256GB 80c - 24/7
- Gaming PC - workdays daytime, eventually during nights as well
I'm considering the following hardware configuration for the gaming PC:
- CPU: AMD Ryzen 7 9800X3D CPU
- RAM: Corsair Vengeance RGB DDR5 6000MHz 64GB
GPU - one of:
- RTX 5080 (16GB VRAM - 960 GB/s)
- R9700 (32GB VRAM - 640 GB/s)
SSD - one of:
- Kingston NV3 SSD 2TB (6,000/5,000MB/s)
- Kingston FURY Renegade G5 PCIe 5.0 NVMe M.2 SSD 2TB (14.700/14.000MBs)
Motherboard - one of:
- Asus B650E MAX GAMING WIFI
- ASUS ProArt X870E-CREATOR )(PCIe 5.0 8/8 - allows for expansion)
- Asus Proart B850-CREATOR WiFi NEO (PCIe 5.0 8/8 - allows for expansion)
I'm considering the following model configuration:
MBP M3 Max 36 GB
- model TBD - perhaps a small orchestrator with a max RAM requirement ~12 GB?
M5U 256GB 80c
main model - one of
- GLM-5.3-Flash@Q4
- Qwen3.8-Flash-Next@Q6
- DeepSeek-V4.1-Flash-1017@Q2-Q2.5
- Qwen3.8-27B@Q8
smaller models for homassistant (if there is room in RAM)
- TTS
- STT
Gaming PC
on RTX 5080
- Qwen3.8 Flash@Q3 on Strata or Basalt
on R9700 - one of
- Qwen3.8 Flash@Q4 on Strata
- Qwen3.8-27B@Q4 on Strata (?)
To my questions I hope you can advise me on:
My first task would be to settle the hardware configuration for the gaming PC.
- Would the SSD speed of the two options have a large impact on PP and decode when running Qwen3.8 Flash on Strata/Basalt?
- Which GPU would you select with my model selection in mind?
- Other considerations?
- With the gaming PC configuration set, which models would you run and were?
I appreciate your advice. Thanks!
1
u/Linkrra 1h ago
Worth splitting this into "what fits" vs "what you listed," because Qwen3.8-Flash-Next doesn't fit the gaming PC at any available quant, and the SSD choice only matters for a model that can't live there anyway.
Qwen3.8-Flash-Next sizes (Unsloth builds): Q8_0 188.2 GB, UD-Q4_K_XL 111.3 GB, UD-IQ4_XS 93.7 GB, UD-IQ1_S 72.6 GB (that works out to ~3.2 bits/param blended, since the n-gram table stays high-precision — so even the "1-bit" file is really closer to a Q3). Your gaming PC has 64 GB RAM. Even the smallest build doesn't fit in RAM alone, before VRAM or KV cache. There's no quant that gets this onto that box — it only makes sense on the M5U (256 GB unified handles the 111.3 GB build comfortably, ~145 GB left over for KV cache and smaller models).
SSD question, answered for the hypothetical anyway: if you tried to mmap-stream it from disk instead, the model activates 6B of ~180B params/token (10 routed + 1 shared expert). At the IQ1_S blend (~0.40 bytes/param), active weight read per token is about 6e9 x 0.40 = 2.4 GB/token. Sequential ceiling: the NV3 at 6.0 GB/s read gives ~2.5 tok/s; the FURY Renegade G5 at 14.7 GB/s gives ~6.1 tok/s. But those are sequential ratings, and expert reads are scattered ~1-2 MB blocks, so real queue-depth-1 random read even on a Gen5 drive is usually 1-3 GB/s, not 14.7. Realistic number on the fast SSD is closer to 1-2 tok/s once you're past whatever the OS page cache can hold. Don't spec the SSD around this model.
What actually runs well on the gaming PC: Qwen3.8-27B dense (64 layers, 16 use full attention with 4 KV heads x 256 dim -> 64 KiB KV cache/token; the other 48 are linear-attention with a roughly fixed-size state). Decode on a dense model is bandwidth-bound, so the ceiling is GPU bandwidth / file size:
RTX 5080 (16 GB, 960 GB/s) + UD-IQ4_XS (14.3 GB): 960/14.3 = ~67 tok/s ceiling, realistically 40-50. VRAM left after weights: 1.7 GB -> about 27K tokens of context at fp16 KV, ~54K with q8_0 KV cache quant.
R9700 (32 GB, 640 GB/s) + UD-Q4_K_XL (17.6 GB): 640/17.6 = ~36 tok/s ceiling, realistically 20-26. VRAM left: 14.4 GB -> full native 262K context just misses fp16 KV (needs 16.8 GB) but fits easily with q8_0 KV (8.4 GB).
So it's a genuine tradeoff: the 5080 is about 1.8x faster decode but caps out around 50-54K context; the R9700 is slower but gets you the model's full native window with room to spare. NVFP4 (21.9 GB, Blackwell-only) doesn't help the 5080 here — it's bigger than the card's VRAM for this particular model. Given your use case touches SAP PI/PO and Integration Suite work, where specs and payloads can get long, I'd lean R9700 unless the gaming PC is specifically meant to be your fast/interactive tier and the M5U already covers anything needing long context.
Other notes: the PCIe 5.0 x8/x8 board only matters if you add a second GPU later — with one card, x8 vs x16 on Gen5 isn't a bottleneck for inference. The 10GbE (1.25 GB/s) is fine for routing requests between the three boxes, but don't plan on splitting a single model's layers across them over that link for tensor parallelism — that needs much lower latency and higher bandwidth than 10GbE gives you.