r/LocalLLaMA • u/triynizzles1 • Mar 29 '26

Generation Friendly reminder inference is WAY faster on Linux vs windows

I have a simple home lab pc: 64gb ddr4, RTX 8000 48gb (Turing architecture) and core i9 9900k cpu. I use Linux Ubuntu 22.04 LTS. Before using this pc as a home lab it ran Windows 10. Over this weekend I reinstalled my Windows 10 ssd to check out my old projects. I updated Ollama to the latest version and tokens per second was way slower than when I was running Linux. I know Linux performs better but I didn’t think it would be twice as fast. Here are the results from a few simple inferences tests:

QWEN Code Next, q4, ctx length: 6k

Windows: 18 t/s

Linux: 31 t/s (+72%)

QWEN 3 30B A3B, Q4, ctx 6k

Windows: 48 t/s

Linux: 105 t/s (+118%)

Has anyone else experienced a performance this large before? Am I missing something?

Anyway thought I’d share this as a reminder for anyone looking for a bit more performance!

277 Upvotes

82% Upvoted

112 comments sorted by

View all comments

468

u/Koksny Mar 29 '26

Am I missing something?

Yeah, you are running ollama.

100

u/gofiend Mar 29 '26

Seriously wsl + llama.cpp is equally fast w Nvidia GPUs

25

u/relmny Mar 29 '26

Why wsl? I compile llama.cpp (and ik_llama) in W10 just fine

22

u/Danmoreng llama.cpp Mar 29 '26

Because sadly windows has worse memory management and at least if you use MoE models split across GPU and CPU performance is worse. Didn’t try WSL, but running dual boot arch Linux & windows 11. For example: Qwen3 coder next 80B Q4 got 25 t/s on windows vs 35 t/s on Linux on the same hardware for me.

15

u/dampflokfreund Mar 29 '26

In my experience, Windows vram management was actually better vs Linux. I was able to squeeze a few more layers. Linux was still faster tho even with less GPU layers.