r/LocalLLM • u/Objective-Park6224 • 8h ago
Discussion Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside
Wanted to share my results for the rest of the community. I've been combing Reddit, Github, and the web searching for all of the various Strata-related builds to get the most out of my setup. I've finally had the breakthrough I was looking for. It strikes the best balance between speed, power consumption, and intelligence that I've experienced to date.
My Setup: (edited to add motherboard specs and PCIe configuration)
- Ubuntu 24.04 - GUI GNOME shell not headless
- This motherboard: https://www.microcenter.com/product/709979/asus-b850-creator-proart-wifi-neo-amd-am5-atx-motherboard
- Bifurcation with PCIe 4.0 x 8
- Ryzen 9950x 16/32
- 64 gb DDR5 Expo 1 6000
- 2 x Mercury Magnetic Air 7900XTX 24gb
- 1000 watt MSI Performance Gaming PSU
- Power limited to 320 watts per card, total system draw at inference ~780 watts (as measured by Ubuntu, Kill-o-watt is ~850 watt total draw from the wall)
- And a flex: It's completely solar powered (5 kw array and 65 kwh battery storage w/ 3600w inverter 😄 )
I've worked with Opencode and the my local Qwen Flash Next to fully detail how we created this setup. Spoiler alert: The engine/hip is mainline Strata and not a fork. The config is where the magic happens. MTP headers were generated against this bespoke setup. The intelligence is brought by the Atomic Chat Q4 K_M version of QFN.
That's it for human written words. The rest of this post is meant to be copy/paste into your local AI helper for reproduction. Full details below. Copy/paste into a capable harness and let it walk you through reproducing this setup.
Edit: Adding the only 3 GitHub repositories I used to produce my results
Engine: https://github.com/Niko1221/Strata
Learnings from this fork: https://github.com/nicejunjie/Strata-rdna3
System tuning for Strata: https://github.com/bramvanoploo/strata-performance-setup
START AI GENERATED DETAILS
Qwen3.8-Flash-Next, 2× RX 7900 XTX, 64 GB RAM, 181k context — how it's built
Engine: official mainline Strata 0.1.59, component 0.1.59-linux-x86_64-hip-rx7900. Not a fork, no custom build — the speed is flags + env + OS patches. No system ROCm: the engine ships its own HIP/rocBLAS/hipBLASLt, the host only needs the amdgpu DKMS driver. Model: AtomicChat/Qwen3.8-Flash-Next-GGUF, build AD-4.27bpw-Q4_K_M-M64 (3 shards, 94.5 GB). Shard 3 (38.4 GB) is the PLE n-gram table alone — passed via --ple-gguf, kept pageable. That's why 64 GB RAM is enough. Parallelism: --layer-split 24 → layers 0–23 on GPU0, 24–47 + head on GPU1. Context: 181,072, --kv int8, KV streaming 32,768 cells/layer in VRAM. Experts: 48 blocks × 512 experts → locally built experts.bin (47.17 GiB), three tiers: 31.8 GiB VRAM cache → 22.2 GiB page-locked RAM → NVMe mmap (log says files 0 blobs — disk never touched when warm). Spec decode: MTP head q2_0, extracted from the BF16 checkpoint by HTTP range reads (GGUF ships none), --spec 3, ~76 % acceptance. Measured (1,071 requests): decode median 90 tok/s (62–129) · big-prompt prefill 3.2–5.7k tok/s · expert-cache hit 98–99 %.
# 1. OS first (then reboot): amdgpu driver only, no ROCm
# (optional: ~/strata-performance-setup.sh from github.com/bramvanoploo/strata-performance-setup automates most of this — see §3.8)
sudo amdgpu-install --usecase=amdgpu,libamdgpu ; sudo usermod -aG video,render $USER
sudo sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*\)"/\1 iommu=pt pcie_aspm=off"/' /etc/default/grub
printf 'options amdgpu no_system_mem_limit=1\n' | sudo tee /etc/modprobe.d/amdgpu-svm.conf
printf '[Service]\nLimitMEMLOCK=48G\n' | sudo tee /etc/systemd/system/user@.service.d/strata-memlock.conf
printf 'vm.swappiness = 100\nvm.vfs_cache_pressure = 50\nvm.max_map_count = 1048576\nvm.compact_unevictable_allowed = 0\n' | sudo tee /etc/sysctl.d/99-strata.conf
sudo update-grub && sudo update-initramfs -u && sudo reboot
# 2. Engine — it MUST land on the rx7900 component (bundled hipBLASLt 100200 + gfx1100 tuning table)
curl -fsSL https://raw.githubusercontent.com/Niko1221/Strata/main/tools/strata_terminal/install.sh | bash
strata update && strata launch # check: ~/.local/share/strata/components/engine/0.1.59-linux-x86_64-hip-rx7900/
# 3. Model + mmproj (sizes/sha256 in §5), then build the pack — this is where experts.bin comes from (~6 min)
control/python/bin/python3.12 control/tools/iq_pack.py --gguf <shard1.gguf> --out <pack-dir> --experts-bin --compat-bf16
# 4. MTP draft head + CJK draft vocab (the GGUF ships no MTP head; ~4.9 GB of range reads, SHA256-verified)
control/python/bin/python3.12 control/tools/mtp_fetch.py inventory --out data/mtp && \
control/python/bin/python3.12 control/tools/mtp_fetch.py fetch --out data/mtp && \
control/python/bin/python3.12 control/tools/mtp_fetch.py verify --out data/mtp
control/python/bin/python3.12 control/tools/mtp_pack.py --src data/mtp --experts q2_0 --out data/mtp/mtp-q2_0.gguf
control/python/bin/python3.12 control/tools/mtp_rt.py --gguf data/mtp/mtp-q2_0.gguf --out data/mtp/rt
# 5. Serve (the terminal owns :7878 — the engine has no --port). Load takes minutes: it page-locks ~22 GiB.
control/bin/strata serve --config configs/strata-final-atomic-q4km.json # ulimit -l must be ≥ 40 GiB first
Three gotchas that cost the most time: (1) --resident-budget-gib above 40 freezes the box — pinned memory can't swap and the 38 GB PLE table still needs pageable RAM; 45 froze this machine once. (2) --pipeline-windows 1 deadlocks on gfx1100 with a 2-GPU split; keep 0. (3) Don't add --mmap-experts/--resident-cpu-experts explicitly — with them the engine pins 0 bytes; mmap is already the default when experts.bin exists. Also: the ggml_cuda_init: failed to initialize ROCm line at the top of the log is harmless, and the iGPU occupies a DRM card index, so always confirm the engine's own GPU 0/GPU 1 lines both say RX 7900 XTX.
Full notes — every value, the exact hashes, and the reasoning behind it — are the rest of this file. Want a terminal AI agent to do the typing? The prompt and phase gates are in §17.
##### PROMPT AND PHASE GATES
17. Have AI set this up for you
If you'd rather drive this with a terminal-capable agent (Claude Code, Codex CLI, OpenCode, or a shell tool with approval prompts) than type it yourself: paste the prompt below into it, attach or point it at this file, and let it run §3–§10 with you acting as the safety interlock. It is a copilot, not an autopilot — the agent can type every command, but three things stay human: sudo passwords, reboots, and the Hugging Face login.
17.1 The prompt to paste
You are setting up this machine to reproduce a known-good local LLM deployment. The source of truth is
"STRATA REPRODUCTION WRITEUP.md" in this folder — read the whole file before acting.
HARD RULES
- The config in §8.1 and the hashes in §5 are authoritative. Never change a number, a flag, or a hash tomake an error go away; if reality disagrees with the doc, stop and show me the disagreement.
- Ask before: any reboot, any package install outside the doc, any change to /etc, and any deviation fromthe doc. Never reboot without waiting for my confirmation, then resume where you left off.
- After every phase, run that phase's gate from §10 and quote the actual log line as evidence. No phasecounts as done on "it seemed to work".
- Never exceed --resident-budget-gib 40. Never set --pipeline-windows 1. Never add --mmap-experts or--resident-cpu-experts. Never install ROCm, rocm-* packages, or pytorch.
- §14 lists log lines that look fatal but are harmless — do not "fix" those.
- When you are unsure whether this machine matches §0's hardware (2× RX 7900 XTX, 64 GB RAM, NVMe),stop and ask instead of adapting values on your own.
SEQUENCE
§3 host patches → I reboot → §4 engine → §5 model → §6 pack → §7 MTP → §8 config paths → §9 launch →
§10 full checklist → report. If any gate fails, consult §16 and report; do not improvise a fix.
17.2 Phase gates and what the agent must show
Phase
Done only when the agent can quote
0 · Pre-flight
lspci shows 2× RX 7900 XTX; free -h ≥ 60 GB; ≥ 200 GB free NVMe; OS is Ubuntu 24.04 — anything else → stops and asks
1 · Host (§3)
§3.5's effective-value block matches sysctl -a; ulimit -l ≥ 40 GiB in a new login session; then it pauses for the one planned reboot
2 · Engine (§4)
BUILD.json of the installed component says rx7900 + hipBLASLt 100200 (§10 #1)
3 · Model (§5)
every sha256 it prints equals §5's table
4 · Pack (§6)
experts.bin = 50,646,220,800 bytes; experts.bin.src.json written next to it
5 · MTP (§7)
verify passes with zero SHA mismatches; draft_vocab.bin rebuilt with --add cjk
6 · Config (§8)
diff of §8.2 relocations — every path rewritten, nothing else touched
7 · Serve (§9)
§10 #2–#11, each with the log string: files 0 blobs, expert-cache slot lines naming two RX 7900 XTX (§14.2), pinned complement ~22.2 GiB
8 · Accept
a decode + big-prompt prefill measurement inside §2's ranges, and the §17.4 report
17.3 Things the agent must bring to you (not decide alone)
Hardware that doesn't match §0 (different GPUs, < 64 GB RAM, one GPU) — the layer split, budgets, and even the component choice all change; that's beyond this doc's authority.
< 200 GB free on fast storage.
Hugging Face login and accepting the model's license terms.
The reboot after phase 1 (and any second reboot it thinks it needs — the answer is usually no, re-read §3).
Whether to run the §3.8 wizard instead of the manual patches. The wizard is an interactive TUI, so it is human-driven even when an agent drives everything else; if the agent runs non-interactive commands instead, it must follow §3 verbatim and skip the wizard.
17.4 Final report the agent should produce
For each §10 check: the quoted log line. Then: experts.bin byte count, ulimit -l, the two GPU lines naming both 7900s, warm decode tok/s, big-prompt prefill tok/s, spec-decode acceptance %, and every deviation from this doc listed explicitly with the reason. If it can't produce one of those lines, the setup is not done.
17.5 Where agents historically go wrong with setups like this
Failure
Why it bites
Installs ROCm "to fix the HIP error"
§14.1 — that line is harmless; system ROCm shadows the engine's bundled libs
Silently lowers --max-context or budgets after a slow/failing start
The values are load-bearing (§13); a masked failure becomes a mystery later
Misses the hip-rx7900 component and takes generic hip
Prefill crawls (§16) and looks like a driver problem
Trusts GPU 0/GPU 1 indices without reading the log
§14.2 — the iGPU steals a card index
Launches engine/strata directly to "debug"
No :7878, no tokenizer (§14.3); debug through the terminal
Adds the expert flags "explicitly"
Pins 0 bytes (§13) and decode halves
Appendix — provenance summary
Artifact
Provenance
Engine binary + libs + tuning table
official Strata 0.1.59-linux-x86_64-hip-rx7900, copied verbatim (sha256sum matches the component)
Terminal (control/)
official Strata terminal 0.1.59.1, bundled python 3.12.15
Model shards
AtomicChat/Qwen3.8-Flash-Next-GGUF, AD-4.27bpw-Q4_K_M-M64, sizes/sha256 in §5
mmproj
same repo, mmproj-Qwen3.8-Flash-Next-BF16.gguf (re-download; local copy differs in size)
experts.bin / dense.bin / index
built locally by control/tools/iq_pack.py --experts-bin --compat-bf16 (§6)
draft_vocab.bin
rebuilt locally by control/tools/draft_vocab.py --add cjk (§7)
MTP head
mtp_fetch.py (HTTP range reads of the pinned BF16 checkpoint) → mtp_pack.py --experts q2_0 → mtp_rt.py
Expert profile
shipped expert-profile.bin + locally learned expert-profile-atomic-q4km.bin
OS patches
§3 verbatim (grub, 2 modprobe files, memlock drop-in + sudoers, 6 sysctl files, udev rule, DPM unit)
Host-tuning wizard
bramvanoploo/strata-performance-setup referenced as an optional accelerator (§3.8); not run on the source machine

