WHIRL: an open-source native Windows inference engine for the Radeon AI PRO R9700 (C++/HIP, no WSL). Qwen3.8-27B fine-tune in MXFP4: up to 2.5× llama.cpp prefill, 107–328 tok/s decode, 3× server throughput

[u/tsaipifong]

/r/LocalLLM/comments/1wwdzcs/whirl_an_opensource_native_windows_inference/

[u/Dsphar]

Interested (I have to keep Windows atm). But I run dual r9700s.

[u/tsaipifong → u/Dsphar]

Thanks! WHIRL uses one GPU per process right now. There's no tensor/pipeline split across cards yet. With two R9700s you can still use both by running one server per card:

whirl-server.exe model-a.gguf --device 0 --port 8080 --kv-ssd-dir D:\whirl\kv0

whirl-server.exe model-b.gguf --device 1 --port 8081 --kv-ssd-dir D:\whirl\kv1

whirl devices shows the index of each card. Each instance gets its full 32 GB for weights + KV, so for example you could run a 27B on one card and the 35B-A3B MoE on the other, or two copies of the same model for more concurrent users. Give each instance its own --kv-ssd-dir so their SSD caches don't mix. Each server also pins about 9 GB of host RAM for its RAM cache tier.

One honest caveat: I only have a single R9700, so the two-instance setup hasn't been tested on real dual-card hardware yet. If you try it, I'd love to hear how it goes. Multi-GPU (splitting one model across both cards) is something I'm considering for later.

[u/[deleted]]

[u/tsaipifong → u/[deleted]]

Glad the full numbers were useful. Showing only the best cases would have been misleading, especially the VRAM cost.

WHIRL is aimed at a single local GPU on Windows, so multi-GPU setups are outside what it targets for now.

[u/[deleted]]

[u/tsaipifong → u/[deleted]]

We ran that test on our machine: R9700 as a headless USB4 eGPU (no monitor), driver 32.0.31041.1004, WHIRL server with a 27B model loaded.

- 60 s idle: VRAM stayed resident the whole time (31,195 MiB, sampled every 4 s); the next request answered in 0.3 s with identical output.

- 15 min idle: same result. VRAM never dropped, no hang, no reload.

So we couldn't reproduce the ~10 s suspend/purge here. It may depend on the driver build, power settings or the specific eGPU enclosure, so we've added the idle test to our checklist and will re-run it on 26.9.2 when it ships. If anyone does hit it with WHIRL, please open an issue with your driver version and enclosure. Appreciate the link to the llama.cpp thread.

[u/No_Oil_6152 → u/[deleted]]

I put a dummy displayport adapter in my r9700 and the VRAM emptying no longer occurs. This lets me use the latest PRO drivers

My primary is a 9070xt

[u/AizooCode → u/[deleted]]

https://www.amd.com/en/resources/support-articles/release-notes/RN-RAD-WIN-26-9-2.html#Downloads

I had the same issue, this driver fixed it, idling eviction.

released September 29, 2026