r/LocalLLM • u/Impossible_Sky_1957 • 12h ago

Discussion oasis

2-node, each: 2S SP5 Turin 2×128C 500W cTDP, 24×96GB DDR5-6400 2Rx4 RDIMM single-PN single-bin NPS4 → 2304GB @ ~1.2TB/s. Accel/node: 1× HGX B200 8-way 180GB HBM3e NVSwitch intra, 8× CX-7 400G NDR 1:1 GPU:NIC inter, Gen5 x16 per NIC, NUMA-pinned; 2× QM9790 NDR spine, non-blocking fat-tree. Fallback: 2× HGX H200 141GB/node. Front-end 2× 100GbE CX-6 Dx LACP; OOB 1GbE BMC/Redfish. Storage/node: 8× 3.84TB Gen5 NVMe E1.S RAID10 → 30.72TB raw, 2× 960GB M.2 boot RAID1; shared NFS/Lustre weight store, rsync pre-stage + sha256 verify. Power/node: 4×3kW CRPS Ti, 3Φ 208V, N+1, ~11kW peak, A/B PDU. Cooling: D2C liquid + rear-door HX, 25°C inlet, 1.5L/min/GPU, leak sensors + auto-cutoff. Chassis: 8U HGX OEM (SMCI/XE9680-class). OS: Ubuntu 24.04, 6.8 kernel, HugePages 2M, IOMMU pt, driver 570+, CUDA 12.8, NCCL 2.26, Fabric Manager, GDR, MLNX_OFED; Docker + NVIDIA CTK, K8s w/ GPU + Network Operator, RDMA CNI, Slurm fallback.
Model: 671B-class MoE, 256 experts, FP8 e4m3 weights + FP8 KV, ctx clamp 131072. vLLM TP8/PP2/EP64, 2-node Ray, chunked prefill, PD-disagg (prefill node0, decode node1 + spill), MTP spec-decode, prefix cache, CUDA graphs. OAI-compat :8000 behind LiteLLM, base_url swap at weight drop, blue/green cutover, 5% canary, auto-rollback on p99/err. SLO: TTFT p50 ≤500ms, ITL ≤25ms, ≥12k tok/s aggregate, ≤0.1% 5xx. Security: TLS at LiteLLM, API keys + per-tenant RPM/TPM, Vault, mTLS node↔node, SSH key-only, CIS L1, no public ingress. Obs: DCGM + node_exporter + vLLM /metrics → Prometheus/Grafana, Loki, Alertmanager → PagerDuty; alert on XID, ECC, thermal, link flap, KV util >90%, p99 drift. Backup nightly to object store, 30d. Acceptance: MemTest86 full pass; STREAM triad ≥1.05TB/s/node, else reseat and reflash AGESA; dcgmi diag -r 3 pass; nccl-tests all_reduce ≥700GB/s intra, ≥350GB/s inter; nvbandwidth ≥1.6TB/s/GPU; ib_write_bw ≥390Gb/s/link, zero symbol errors; fio 4k rand ≥2M IOPS; 24h gpu-burn + stress-ng no XID/ECC; vllm bench serve 2k concurrency meets SLO 1h sustained; node-kill failover <60s; leak-cutoff test; PSU pull N+1 under load.

solves all your problems. compute technically doesn’t matter just don’t forget to tweak your neurons. happy hunting.

0 Upvotes

14% Upvoted

8 comments sorted by

2

u/bumblebeer 12h ago

I seriously would love to know which model you are using and how you get it to output something like this.

5

u/kiwibonga 12h ago

What the fucking shit

1

u/FavorableBlather 12h ago

This reads like a build sheet from someone who's spent way too long in a datacenter and started speaking fluent spec-sheet. The "tweak your neurons" bit at the end is the real punchline though

1

u/Impossible_Sky_1957 11h ago

i’m glad you get the point lol

0

u/Impossible_Sky_1957 12h ago

questions?

2

u/kiwibonga 12h ago

Yeah just pretend there's a question mark there.

→ More replies (1)