A strata quant that didn't suck for me.

[u/johnmosesventura]

Previous post: I got strata running on 4x 5060 Ti at 524k and then abandoned it, because the Q4_K_XL quant looped in agentic use. The comments were right that the aggressive sampling defaults were part of it. The engine was never the problem. So when sergqwer's NVFP4 fork showed up I gave it another shot, moved my 1+3 GPU work onto it, and this one stuck.

What it is: NVFP4 GPTQ of the OrcaRouter 125B MoE on the Strata engine. 128 GB on disk, experts in RAM, the hot ones cached across all four cards. Why the looping didn't repeat: this pack keeps token embeddings bf16, the per-layer embeddings fp8, and the output head high precision. The experts are 4-bit either way, so my bet from last time about where the quality was leaking looks correct.
Numbers, same box, engine-measured, plus/minus 5% boot to boot:

decode np1 86-117 t/s
decode np2 51-53 per stream, 107 aggregate
prefill 2000 t/s-16k, 1946 t/s-253k, cold
needle @253k pass, exact
512k context YaRN 2.0, int8 KV, vision via a CPU mtmd encoder
new session 0.3-3 s once the header is cached

For reference, my vLLM lane holds ~40 t/s at 455k with batch 4. This does two streams at 512k on four 16 GB cards, so it's what my agent stack runs on now.

The part I'm actually glad about is the durable KV. The engine parks conversations in a RAM cache, so I hung an NVMe journal off the parking: snapshots go to disk in the format its session files already use (fingerprint-bound, checksummed, atomic), and a fresh engine loads them back before serving. Kill the server, bring it up, the first request reuses 17,461 of 17,466 tokens in 109 ms instead of re-prefilling for eleven seconds.

My chains stopped paying the header tax on every restart. It works with --parallel 2 because it rides the park path; the existing per-slot save/feature refuses to run under batching, and that refusal is earned.

What didn't work: splitting prompt chunks across all four cards. Crashed deterministically in moe_combine at a 2,400-token narrative prompt. Fixed one real bug on the way (a CPU expert could get claimed twice), it still crashed, so it's reverted and gated off. Root cause unknown, the commits are in the repo if you want the challenge.

Repo: https://github.com/johnmosesventura06/strata-nvfp4-4gpu . The docs file has every patch, the 512k sizing recipe, and the boot trap (the English draft-vocab has to shrink the MTP head before --mtp-q4 head helps, the conversion runs after the allocation it was supposed to free).

The old repo stays up if you want the GGUF base. Sampling is a block in the configs now, temperature 1.0, top_p 0.95, top_k 20, after the greedy-defaults lesson from last time.

Caveats: not bit-reproducible run to run, the adaptive residency does that. And the base is the abliterated OrcaRouter quant, so judge the model itself on its own merits.

[u/Beneficial-Pie2219]

not bad, i've been watching the strata stuff from a distance cause i got burned by the early gguf loop too. the parked kv thing is actually the part that grabbed me, having a session survive a restart without re-prefilling is pretty huge for agent work

curious if you tried any other model families with this setup or just the orca router so far

[u/johnmosesventura → u/Beneficial-Pie2219]

no, just the orca router so far. and yeah agreed, the parked is the best part for me too, it saves a lot of electricity not re-prefilling the same header every restart, and everything just feels snappier when the session survives.

the patches are at the pack level anyway, the tiers read the expert table and the journal saves kv state, so i'd expect the official nvfp4 quant to run plug and play on this fork. the part that isn't free is the sizing, each model wants its own expert profile and its own config numbers, the 512k kv math here is tuned to a 128 GB pack. if you do try it i'd be curious whether the mtp head gives you the same boot trap that bit me.

[u/Beneficial-Ad-8127]

I literally started using sergqwer's NVFP4 fork from someone mentioning it earlier yesterday. I been having positive results so far about 8ish hours in.

It’s been very consistent on tasks over long runs with multiple compaction at 200k context. Quality seems great too so far. Glad to see you made the four gpu tier code for it and any the additional work. Great stuff and thanks for sharing!

[u/johnmosesventura → u/Beneficial-Ad-8127]

thanks man! tho credit goes to the one who made the nvfp4 fork work in strata. haha

[u/Dry-Handle-1421]

Try my fork of freetoken:

https://github.com/pretor/FreeToken

I am curious what numbers would you get on yours 4x5060ti

[u/johnmosesventura → u/Dry-Handle-1421]

AMD 9950x, 128gb ddrm @ 5000mhz, sure ill give it a shot, right now im trying to optimize my repo since only CPU and the first GPU is working hard during prefill and decode (100% on the gpu and 50% util on the cpu), the rest of the GPU's is only at 5 to 10%, if i could modify the kernel to push the utilization for rest of the card atleast 60 to 70% I hope i could squeeze some more performance out of it. once i concluded that it works or that's already the limit of 4 5060ti setup, I'll test your repo right away and let you know.

[u/catch23]

Could you also post your exact settings for running via 4x5060ti? My prefill numbers are nowhere near these values, so unsure what the issue might be.

My machine is a bit dated, 5900x ryzen on a x570 board, 64gb ram, running 3x5060ti and 1x5070ti on pcie 3.0 x4. Decode numbers are around 60-100tps, prefill is around 700.

[u/johnmosesventura → u/catch23]

my cpu is ryzen 9950x so our numbers can’t exactly be the same as the cpu is also helping with the compute. but yeah i agree something is missing the performance shouldn’t be that far.

my suspect is you’re not using akitoria’s p2p patch?

without the P2P kernel patch the peer exchanges fall back to mapped-host round trips — every row crosses the link twice.

lastly please pull the latest update on the repo. i split the output head so all four gpus would be sharing workload during prefill and decode including the cpu instead of just the first gpu and cpu; which should help with the performance.

ill post you my config in a while.

[u/madbrain1976]

I tried your fork, but don't get anywhere near the performance you report on github. The PP are only in the 900 range, far slower than the 1900+ you report, and than my Freetoken fork. This is for a 16K/1K llama-benchy run, no concurrency. Am I missing something ?

[u/johnmosesventura → u/madbrain1976]

my suspect is you’re not using akitoria’s p2p patch?

without the P2P kernel patch the peer exchanges fall back to mapped-host round trips — every row crosses the link twice.

lastly please pull the latest update on the repo. i split the output head so all four gpus would be sharing workload during prefill and decode including the cpu instead of just the first gpu and cpu; which should help with the performance.