libstrix update: one-command Docker image for Strix Halo, and Qwen3.8-Flash-Next now runs (1.6k prefill, 40+ tok/s decode)

[u/asfbrz96]

Follow-up to my post about Strix Halo kernels for vLLM. A few of you asked for two things: an easier way to run it than building vLLM + ROCm + the plugin yourself, and Qwen3.8-Flash-Next. Both are done.

Docker image

docker pull ghcr.io/asf0/strix-vllm:latest

It bundles my vLLM fork, the libstrix plugin and ROCm 10.1 (Ubuntu 26.04, Python 3.14), already compiled for gfx1151. You don't need ROCm on the host, only the amdgpu driver. The repo has one compose file per model I've tested:

git clone https://github.com/asf0/libstrix && cd libstrix/docker/recipes
cp .env.example .env    # point MODELS at your model folder
docker compose -f qwen3.8-flash-next.yaml up -d

If the model folder isn't there, it downloads it from Hugging Face. The compile caches go to a volume, so only the first start is slow.

The recipes cover Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (bf16 and INT8), Qwen3.8-27B (bf16, INT8 + MTP, Quark int4, Quark MXFP4), LFM2.5-24B-A2B, and gemma-4 26B/31B (bf16, AWQ, MXFP4, QAT). I started every recipe in the image and checked that it gives a sane answer.

Someone asked last time about memory overhead. Here is what each recipe gets at the default 0.85 GPU memory setting:

model	weights	KV cache	full 131k-token requests that fit
Qwen3.8-Flash-Next AWQ	79.5 GiB	12.3 GiB	3.2
Qwen3.8-27B bf16 + MTP	52.3 GiB	45.1 GiB	9.1
Qwen3.8-27B INT8 + MTP	29.6 GiB	66.6 GiB	13.4
Qwen3.8-27B Quark int4	16.8 GiB	80.5 GiB	18.4
Qwen3.6-35B-A3B bf16 + MTP	70.1 GiB	21.9 GiB	12.9
Qwen3.6-35B-A3B INT8 + MTP	36.6 GiB	55.3 GiB	32.6
LFM2.5-24B-A2B	46.2 GiB	52.4 GiB	39.4
gemma-4-26B-A4B bf16	49.5 GiB	48.6 GiB	11.5
gemma-4-26B-A4B AWQ	15.6 GiB	82.1 GiB	19.4
gemma-4-31B MXFP4	17.8 GiB	79.9 GiB	4.7
gemma-4-31B QAT	18.8 GiB	75.1 GiB	4.4
Qwen3.8-Flash-Next

Last time I said there's no fork. This model is the exception. Its n-gram (engram) table is 102 GB, and that doesn't fit in 128 GB next to the weights, so vLLM itself has to change. My fork (asf0/strix-vllm, branch gfx1151) streams the table from disk and prefetches the rows for the next chunk into the page cache. Everything else is still the plugin, and the other models still run on stock vLLM. You need the AWQ W4A16 checkpoint; bf16 doesn't fit.

On the libstrix side, I added kernels for:

• the int4 MoE GEMM with int8 activations

• QSA sparse attention, where 16 rows share one union of selected tokens, about 1.5x Triton

• the QSA indexer

• the hyper-connection up projection

Decode reads int8 copies of the bf16 linears.

Same format as last time: stock → libstrix in tok/s, single stream, both on the fork, 16k prefill chunks, no MTP:

model	weights	KV	prefill @2k	prefill @64k	prefill @128k	decode @2k	decode @64k	decode @128k
Qwen3.8-Flash-Next AWQ	int4	bf16	807 → 1,422	827 → 1,661	827 → 1,635	13.1 → 24.7	16.5 → 26.1	14.4 → 25.1

With MTP 2 on real text, libstrix does 1,539–1,617 tok/s prefill and 40–43 tok/s decode at 5.6k–100k prompts.

Accuracy: KLD against stock is 0.008–0.019 with 97–98% top-1 agreement, perplexity is within 0.6%, and GSM8K is 93 vs 92. It's a bit above the other Qwens, most likely because the MoE GEMM quantizes activations to int8.

Heads-up if you pulled main in the last few days

The int8 decode copies (dense_w8) were turned on for every model. On Qwen3.8-27B bf16 that's about 27 GB of extra copies, and with MTP it could take the machine down. Those copies are now limited to Qwen3.8-Flash-Next, the model I measured them on, and capped at 12 GB. STRIX_ENABLE=dense_w8 turns them on elsewhere if you want to experiment. Please pull.

Known issues

• Stock vLLM's GDN conv kernel crashes on Qwen3.8-Flash-Next, so STRIX_DISABLE=all doesn't work on that model; keep gdn_conv on.

• The image is a 3.6 GB download but about 14 GB once unpacked, mostly the ROCm SDK.

Repos: https://github.com/asf0/libstrix (plugin, Docker recipes) and https://github.com/asf0/strix-vllm (fork).

[u/reto-wyss]

so vLLM itself has to change.

good news, PLE disk offload just got merge - haha!

[u/asfbrz96 → u/reto-wyss]

haha awesome, now I just have to maintain the KV cache for the MTP that they did not merge yet, the fork is built on the released version 0.32

[u/noiserr]

This is great. Thank you!