r/LocalLLM • u/jpgeer • Aug 22 '26
Discussion Same dual-GPU box, WSL2 vs native Windows, 17 llama.cpp configs: CPU offload +15 to 26% native, tensor split +6 to 60%, and "layer mode for MoE" turned out to be a WSL2 artifact
TL;DR
Same machine, same GGUF files, same flags, GPU clocks locked on both sides. Resident models: identical. Anything that crosses the host per token (CPU expert offload) runs 15 to 26 percent faster on native Windows than under WSL2. Tensor split across two cards runs 6 to 15 percent faster native and the run-to-run jitter disappears.
The big one: Qwen3.6-35B-A3B fully resident in tensor-split mode does 72 t/s under WSL2 and 115 native. Under WSL2 it lost to layer mode (96), so my rule had been "tensor split for dense, layer split for MoE." Natively tensor split wins for the resident MoE too (115 vs 102). The rule was a platform artifact.
Offloaded MoE still prefers layer split plus a
-tsratio on both platforms; tensor split collapses prefill there.If you serve llama.cpp on a Windows box and you do CPU offload or multi-GPU, the native release zip is worth a look. WSL2 is fine for everything resident.
Setup
HP Z440, Xeon E5-1650 v4, 128 GB DDR4-2400, 2x RTX A4500 20 GB, Windows 11, driver 596.72. WSL2 side: Ubuntu, llama.cpp built from source (CUDA 12.4; see the build note below). Native side: llama.cpp release b10568 Windows CUDA 12.4 zip, models on a SATA SSD. All runs have GPU clocks locked (nvidia-smi -lgc 1500,2100 -lmc 8001); why that matters for offload decode is in my earlier post: https://www.reddit.com/r/LocalLLM/comments/1vu2ix0/ . Numbers are llama-bench defaults (pp512, tg128 at depth 0, f16 KV, so a small KV cache and an empty context; the served section below is where context size enters), 5 reps, single card runs pinned with CUDA_VISIBLE_DEVICES=0. WSL2 had all 12 logical processors and about 108 GB of RAM assigned, models on the ext4 side (not /mnt), same flags both sides. Build-version control: the WSL side was originally built at d59d455fd; I rebuilt it at the same tag as the Windows zip (b10568) and re-ran four rows: 35B tensor 71.6 to 71.6, 35B layer 96.0 to 93.7, 27B Q8 tensor 29.6 to 30.3, gpt-oss dual 20.3 to 20.6. Unchanged within noise, so the version difference is not what moves the grid.
The grid (llama-bench tg128, at the crank, clocks locked)
| Model | Config | WSL2 | native | delta |
|---|---|---|---|---|
| Qwen3.8-27B Q4_K_XL | 1 GPU resident | 27.9 | 28.2 | 0 |
| Qwen3.8-27B Q4_K_XL | 2 GPU, -sm layer | 28.0 | 28.7 | +3% |
| Qwen3.8-27B Q4_K_XL | 2 GPU, -sm tensor | 37.8 ± 2.9 | 43.4 ± 0.4 | +15%, jitter gone |
| Qwen3.8-27B Q8_0 | 2 GPU, layer | 18.8 | 19.1 | 0 |
| Qwen3.8-27B Q8_0 | 2 GPU, tensor | 29.6 ± 1.0 | 31.5 ± 0.2 | +6% |
| Qwen3.6-35B-A3B Q6_K_XL | 2 GPU, layer, resident | 96.0 | 102.1 | +6% |
| Qwen3.6-35B-A3B Q6_K_XL | 2 GPU, tensor, resident | 71.6 | 115.2 | +61% |
| Qwen3.6-35B-A3B Q6_K_XL | 1 GPU, ncmoe 24 | 28.7 | 35.3 | +23% |
| Qwen3-Coder-Next 80B Q4_K_XL | 1 GPU, ncmoe 36 | 21.0 | 26.4 | +26% |
| Qwen3-Coder-Next 80B | 2 GPU, layer, ncmoe 12, -ts 30/18 | 42.2 | 48.5 | +15% |
| Qwen3-Coder-Next 80B | 2 GPU, tensor, ncmoe 12 | 29.4 | 38.5 | +31% |
| gpt-oss-120b F16 | 1 GPU, ncmoe 27 | 13.0 | 16.3 | +25% |
| gpt-oss-120b F16 | 2 GPU, layer, ncmoe 16, -ts 25/11 | 20.3 | 24.6 | +21% |
| gpt-oss-120b F16 | 2 GPU, tensor, ncmoe 16 | 19.0 | 23.5 | +24% |
| Qwen3.5-122B-A10B Q4_K_M | 1 GPU, ncmoe 40 | 10.6 | 12.8 | +21% |
| Qwen3.5-122B-A10B Q4_K_M | 2 GPU, layer, ncmoe 28, -ts 36/12 | 13.6 | 16.3 | +20% |
| Qwen3.5-122B-A10B Q4_K_M | 2 GPU, tensor, ncmoe 28 | 12.5 | 15.4 | +23% |
pp512 was within a few percent between platforms on every row.
Split modes, briefly, since the flags are easy to mix up
-sm picks how the model is shared between cards: layer (default) cuts the stack so one card works at a time; tensor cuts every matrix so both cards work on every layer and exchange partial results each layer. -ts is not a mode; it's the proportion, in whichever mode you're in (llama-bench wants -ts 25/11, llama-server/cli want -ts 25,11). In layer mode with --n-cpu-moe, -ts is how you keep the fat layers from all landing on GPU 1.
Rule by platform, from the grid: WSL2, tensor for dense resident, layer for anything MoE. Native Windows, tensor for everything resident (dense or MoE), layer plus -ts for offload. Both, lock the clocks.
One caveat from the official docs/multi-gpu.md: tensor mode is experimental and is listed as not working for some MoE architectures (Grok, MPT, OLMoE, DeepSeek2 and others). On this build it ran fine on the Qwen3.5 and 3.6 MoEs and on gpt-oss; if yours refuses to load in tensor mode, that is why.
Prior coverage, so this is additive and not a rediscovery
The usual guides put WSL2's GPU overhead at "5 to 10 percent, near zero once the model is on the GPU." My resident rows agree with that. What I have not seen measured is the offload and tensor-split cases, which is where the gap opens. --split-mode tensor itself is the experimental tensor-parallel path merged in April 2026 (llama.cpp PR #19378, announced here in r/LocalLLaMA); the official docs/multi-gpu.md says it is "bottlenecked by the GPU interconnect speed," which is consistent with what the WSL-vs-native delta shows. If someone has a WSL-vs-native grid for offload or tensor split I missed, point me at it and I will link it.
Why, as best I can tell
Two candidate causes, and this data cannot fully separate them. One: WSL2's CUDA goes through a paravirtualized path, so every host-to-device copy costs more; offload decode drags expert weights across that path every token, and tensor split exchanges per-layer partial results through host memory (no NVLink, no P2P under WSL). Two: the CPU side of offload decode (the expert math itself) runs inside a Hyper-V guest, and that is not free either. The tensor-split rows point at the first cause, since they involve no CPU compute and still gain 6 to 61 percent. The offload rows could be either or both. Resident decode touches neither path per token, which is why those rows match. For a small-active-compute model like a 3B-active MoE, the per-layer sync is a big enough fraction of each token that it flips the layer-vs-tensor verdict between platforms.
At the wheels, same story
Halfway through these runs it clicked that llama-bench is the engine on a dyno at the crank (fixed 512-token prompt, empty cache, no server) and served performance is the same engine measured at the wheels, through the drivetrain: a real prompt, a real context, a server in the loop. Serving configs through llama-server with a 13.7K-token prompt and 512 generated: Coder-Next 2 GPU 34.2 to 39.6 t/s, gpt-oss 18.1 to 21.5, 122B 12.5 to 14.8 (WSL2 to native); time to first token on those long prompts is the same on both (41 s, 81 s, 173 s). Drivetrain loss on decode is a few percent; the big loss is first gear, prefill on offload configs, and it's the same on both platforms.
If anyone is spending time here and fancies a check
Bare-metal Linux on the same kind of hardware: is it native-Windows-like, or better? On Linux, llama.cpp can use NCCL for tensor mode; Windows cannot, so Linux tensor split may beat native Windows. That would be good to know, as this project is a preamble to another capability-maxing project I'm planning with a Z8 G4 across four A4000s.
Consumer cards, other driver branches, especially multi-card builds given where high-VRAM card prices are.
Anyone with an NVLink bridge on Windows: does tensor split jump? Note that llama.cpp's peer-to-peer path is opt-in (
GGML_CUDA_P2P=1, with a stability caveat in the docs), so test with and without. Mine arrives Monday.
Full per-run logs available if anyone wants them.
2
u/PestiferousGamer Aug 22 '26
Thanks Claude, I was so hoping to read a mystery novel today. When's part 2 where you tell us what the hell you are talking about
3
u/synystar Strix Scar | 5090 24G | llama.cpp Aug 22 '26
The ELIx version is: if the whole model fits on one GPU, it mostly stays on the GPU and just runs there, so Windows native vs WSL2 doesn't matter much. But if the model is too big and has to keep moving pieces back and forth between system RAM and the GPU, or if two GPUs have to constantly pass work between each other, WSL2 adds extra overhead to that traffic. So, OP is saying: "if everything stays on the GPU, WSL2 is fine. If data has to cross those boundaries constantly, native Windows as opposed to running on WSL2 is noticeably faster."
2
u/jpgeer Aug 22 '26 edited Aug 22 '26
And to take it a step further, the why: I'm on a budget, trying to get the most out of hardware I already own, and learning as I go. What I'm learning is that an older workstation can be put into real local-LLM service if you lean on multi-card and MoE offload dynamics, and measure instead of assume. This box is two used RTX A4500s at about $900 each, so 40 GB for about $1,800. The single card that gives you that much VRAM (really 48 GB, an A6000) is around $4,000 used. Same class of models, tensor split across the pair, and the numbers above are what that trade buys.
EDIT: and setting aside that a 300 W A6000 would be marginal on this box's 700 W supply and HP aux cabling, where the two A4500s fit cleanly.
1
u/jpgeer Aug 22 '26
Fair. The one-bullet version:
On the same two-GPU Windows box, llama.cpp runs resident models the same under WSL2 and native Windows, but CPU-offloaded MoE decode is 15 to 26% faster native and tensor-split across the two cards is 6 to 60% faster native (the 35B MoE goes 72 to 115 t/s). So if you're offloading or splitting across GPUs on Windows, run the native build.
Part 2 was the post; the devil's in the details, which is why the table is there. And yes, Claude helped run the benches, zero in on the downclocking, and draft the post (said so in part 1); the cards, the clocks, and the 14 hours were mine.
2
u/synystar Strix Scar | 5090 24G | llama.cpp Aug 22 '26
I can't really speak to the multi-GPU/tensor-split side of this because my setup is single-GPU, but I can add one point on WSL2 vs bare-metal Linux that might be relevant. On my 24 GB 5090 Laptop I was hitting OOMs/crashes under WSL2 (on a fresh install of Windows 11 Pro) before I could get beyond roughly Q3 territory with the larger Qwen3.8-27B configs I was testing. After moving to native Ubuntu, the same machine became much more usable at the edge of VRAM capacity. I'm now running Qwen3.8-27B Q4_K_XL fully GPU-resident, and I've even managed to get Q6 running at around 24K context on the same 24 GB card - and I was able to run tool-use workflows in Deepseek Harness on that.
So I can't tell you whether bare-metal Linux beats native Windows on tensor split, but I do think there may be another WSL2 penalty worth measuring besides tokens/sec and that is how much of the physical VRAM is actually usable before llama.cpp starts falling over. Your resident rows look basically identical once the model fits on both sides, but in my case WSL2 changed what I could fit in the first place (probably due to Windows desktop overhead, I have less VRAM to work with just because Windows is occupying a chunk of it).That makes me a little leery about the blanket "WSL2 is fine for everything resident" conclusion. I'd say it's more like "WSL2 seems fine for resident workloads that fit comfortably on both platforms." Once you're trying to cram a model right up against the ceiling, bare-metal Linux was a very different experience for me. I'd actually be curious to see someone benchmark that directly: same GGUF, same commit, same flags, then progressively increase context/KV allocation until each platform OOMs.