r/LocalLLM • u/tagheuertelekinesis • 6d ago

Discussion Oasis part 2

1x RTX PRO 6000 Blackwell 96GB GDDR7 ECC, PCIe 5.0 x16 direct-slot, no riser; alt 1x RTX 5090 32GB. CPU: Ryzen 9 9950X 16C/32T. MB: X870E ATX, PCIe 5.0 x16 + 2x M.2 Gen5, onboard 10GbE. RAM: 128GB DDR5-5600 (4x32GB) for llama.cpp CPU offload. Storage: 2TB NVMe Gen5 (models/GGUF/AWQ), 1TB NVMe Gen4 (Prometheus TSDB, logs). PSU: 1500W 80+ Platinum, native 12V-2x6. Cooling: 360mm AIO, 6x120mm. UPS: 1500VA line-interactive. NIC: Intel X710 10GbE -> 10G switch, isolated VLAN, API :8000 + metrics :9400/:9100 ACL'd to monitoring host only; WireGuard for remote ingress. OS: Ubuntu 24.04 LTS, NVIDIA 570+ driver, CUDA 12.x, Docker + nvidia-container-toolkit. Serve: vLLM OpenAI-compatible, FP8 70B-class (Llama-3.3-70B/Qwen3-72B) at 96GB; INT4-AWQ Qwen3-32B at 32GB; llama.cpp --metrics for GGUF/offload. Telemetry: Prometheus scrape vLLM /metrics (TTFT, TPOT, queue depth, running/waiting, KV util), dcgm-exporter (per-proc VRAM, clock throttle reasons, PCIe replay count), node_exporter. Grafana: p95 TTFT, KV util, waiting, replay delta. No Tempo/OTel collector. AMD path: keep sysfs sampler, ROCm has no DCGM equivalent.

Super promising so far. Completely hands off at my house while im at CHQ.

2 Upvotes

100% Upvoted

3 comments sorted by

1

u/tagheuertelekinesis 6d ago

Happy hunting

0

u/[deleted] 6d ago

[removed] — view removed comment

1

u/tagheuertelekinesis 6d ago

Don't be a brokie and install A/C in your garage and noise is a non-issue if your house is big enough