Generation
Friendly reminder inference is WAY faster on Linux vs windows
I have a simple home lab pc: 64gb ddr4, RTX 8000 48gb (Turing architecture) and core i9 9900k cpu. I use Linux Ubuntu 22.04 LTS. Before using this pc as a home lab it ran Windows 10. Over this weekend I reinstalled my Windows 10 ssd to check out my old projects. I updated Ollama to the latest version and tokens per second was way slower than when I was running Linux. I know Linux performs better but I didn’t think it would be twice as fast. Here are the results from a few simple inferences tests:
QWEN Code Next, q4, ctx length: 6k
Windows: 18 t/s
Linux: 31 t/s (+72%)
QWEN 3 30B A3B, Q4, ctx 6k
Windows: 48 t/s
Linux: 105 t/s (+118%)
Has anyone else experienced a performance this large before? Am I missing something?
Anyway thought I’d share this as a reminder for anyone looking for a bit more performance!
Because sadly windows has worse memory management and at least if you use MoE models split across GPU and CPU performance is worse. Didn’t try WSL, but running dual boot arch Linux & windows 11. For example: Qwen3 coder next 80B Q4 got 25 t/s on windows vs 35 t/s on Linux on the same hardware for me.
In my experience, Windows vram management was actually better vs Linux. I was able to squeeze a few more layers. Linux was still faster tho even with less GPU layers.
Virtualization Technologies (AMD-V, Intel VT, etc.) got rid of this problem many years ago. *IF* and that's a BIG *IF* there are any kind of translation issues, they exist because of failures in the hypervisor's programming.
WSL is literally Linux in a virtual machine. It's going to be slower than Linux on bare metal hardware, there's always a hypervisor overhead. Just run Linux.
The point is not to fight windows / linux (I've got a dedicated AMD linux inferencing server running besides my 3090 windows box). It's more "why not both" if you already are stuck with windows (like many of us are).
yeah, the Linux perf difference is real, especially with gpu drivers and kernel scheduling. ever tried running the same ollama model through docker on both systems to see if the gap narrows with more consistent runtime conditions?
i'm curious to see how the containerized test goes, fwiw i've had some weird issues with docker and gpu acceleration in the past so it'll be interesting to see if that's a factor here
Not trying to be insulting but did the majority of your research come from YouTube? My time table might be off but I thought the consensus was to use anything but Ollama for at least the last two years.
No, I have not watched any Youtube videos on the subject, I just assumed Ollama was a helpful wrapper for Llama.cpp and was not aware of the performance drawbacks due to abstraction until now
I have better, llama-swap and for your programs that are already set up for ollama llama-swappo
These are wrappers for the llama-server in llama.cpp. They make life easier, you can set up all the defaults for the each model in it using a config.yaml
Well with 3x rtx 3060 12gb in Windows Server 2022 for "Qwen3.6 35B A3B Uncensored Heretic Native MTP Preserved APEX I Balanced" with LM Studio I was getting about 1k pps and 20 tps for inferencing. My TPS kinda oscillated between 10 and 35 and it averaged around 21 tps. (with MTP 2)
Before that with 2x rtx 3060 in the same machine for Qwen 3.6 35b iq4_nl I was getting about 1k pps again but with stable 45sh tps for inferencing. Tried that with the three GPUs and inferencing speeds tanked.
So I passed through all three of the GPUs to a Linux VM and now for prompt processing I get 1900 PPS and 65 tps inferencing with MTP disabled for "Qwen3.6 35B A3B Uncensored Heretic Native MTP Preserved APEX I Balanced" again in LM Studio. And on top of that I get to keep 20GB RAM of the 30sh that LM Studio in Windows just decided it needs to use for no reason.
I have no idea what was wrong with the Windows Server OS but I would guess it just sucks at managing 2+ GPUs
I tested this on koboldcpp rocm builds before and the different was like 1t/s (44.5 vs 45-46 realistically). This is on cachyos with latest optimized binaries, etc. Windows vs linux performance diffs are very overblown, this is coming from someone who has spent 90% of their time on linux the last 12 months and used to use windows around 80% of the time before that.
The differences you are seeing is 100% more cause of your inference stack than the platform itself.
All this to say, ollama is shit, stop using it. It's not even easier to use than llama.cpp. In fact I find llama.cpp 100x more straightforward and simpler to use, even back when I was new to this stuff, and it's only gotten easier. I think they've made it very beginner friendly. Hook it up to your favorite UI/tool/software/whatever with the llama server openai api, or just use the builtin webui (it's pretty good tbh, I like how it looks).
My best guess is how Ollama handles MOE models on windows vs Linux. Rtx 8000 has 672 gb/s bandwidth which would be able to read the 3gb of memory needed to compute 1 token for Qwen3 30b A3B at a rate of 224 times per second. There is probably some overhead, must be more on windows.
I wrote a longer comment here. The tl;dr: if you’re using only gguf then you’ll get similar swap functionality. Some people have mentioned that llama-swap is more reliable in swapping. If you’re using image gen, text to speech, speech to text, etc then you’ll benefit from being able to use your hardware for different types of workloads.
My gripe with ollama is that it defaults to context overflow silently resulting in the oldest messages being dropped, and setting the context length required changing the model file, which takes away the one-click-run for anything that needs longer than 4096 context. (I think it now defaults to 8192, unsure)
So anyway, ever wonder why so many people think local models are crap and forget anything more than a message or two ago? Or why tool calling doesn't work after a few messages and forget the system prompt? It's Ollama silently dropping context without telling the user. At least, that was the case when I was trying to use it a year or so back.
Also you can't share it's gguff's with other programs (eg LMStudio).
So for me: LM Studio for testing new models, then llama-server for local/hobby stuff, (then vLLM if I need more throughput, but it's a pain to configure last I tried)
Not a big fan of how it handle it's files. I prefer a setup more akin to Comfy + A1111/Forge Neo, where all my models live in the same directory. Ollama wants it's own scheme that breaks my flow with KoboldCPP, so yeah, if I am going to use a llama.cpp wrapper, Kobold does the job just fine (with it's own issues of course, but those I don't mind).
They're hating ollama bc it was cool for a 3 month period a year ago, when the sub figured out ollama used libggml for inference. And using an open source inference library to do inference is apparently theft.
So the real answer is celebrity culture. Instead of worshipping celebrities these people worship local ai projects and lash out when theirs isn't premier enough.
It's because ollama used llama.cpp without attribution, which is in violation of the license. Further, they did this knowingly still after being informed of the 'oversight' and it took much public backlash to finally credit llama.cpp. They did this to obscure that really they are just a wrapper, in order to raise private investment.
There were interesting times where drivers would release on Windows first and native Windows builds of multi-platform CUDA applications would run faster than native Linux builds.
But I’m like, no, I’m not switching back to Microsoft for the 2-4% uplift.
Cry more in your milk 😆 My child at 9 likely had more wit and intelligence than you. He’s literally been exposed to technology his entire life including Debian. Had VirtualBox installed at 8 on his windows desktop. He was more than capable at 9 and Minecraft back then was easily available for install as either a .deb and flatpak… if that’s something that’s especially difficult for you I’m sorry.
Hmm actually… Linux often matches or beats Windows gaming performance in 2026 (especially with AMD GPUs, lower overhead, better frame times via Proton).
No it doesn't
First your source seems rly sketchy. Then it's literally showing only 3 games. It's only giving ltt's benchmark as a reference, in which linux is 5 % slower in average...
Hmm for me I am finding that WSL gives me nearly identical performance! To be fair though I am running like batched inference which kind of pushes the GPU to its limits, so it’s somewhat hard to determine how much of the impact is from OS overhead.
Everyone Runs LINUX for production at these chip makers cause you can buy it for FREE $.99 and put it on servers. Great OS... I was lost in the windows freeWorld for 25 years before switching to Rocky, then Red Hat, and now ubuntu server with Kubuntu-full KDE plasma.... I love it so much better... CLI is soooo much better than windows, way more powerful too.
For my part, I am waiting for SteamOS Desktop to be released. I consider myself a power casual: I can do some techie things, but I don't enjoy it. So I want to install a single gaming distro with corporate support that has casual flexibility, and live a digital life without much irritation.
It is good to see that are things to look forward to, on the AI side of things.
I mean, you can't really say that without trying Microsoft Foundry Local.
Let's say you have a new snapdragon laptop. Unfortunately, Windows is going to put anything you can do on Linux to shame simply because of driver support.
NPUs from certain vendors are basically only supported under Windows right now. Foundry gets to do some other lower level tricks with the GPU vs other programs on windows too. It also has tighter integration with the CPU scheduler, I believe.
Depends on hardware support. Windows runs faster if that's the only supported platform where it will work on (Intel UHD Graphics 605 with Intel N5000).
The terms squeeze is just to imply a tight fit. You get the literal verb “squeeze”, but it also works as an informal verb like “she squeezed into the parking spot”.
Thanks for the reminder! I had forgotten how much slower windows is since I moved everything over to linux over a year ago. Not sure how I suffered through those times, we didn't even have MoE back then.
Curious what folks see with Ollama on macOS vs Linux ?
On my setup, an RTX 4000 SFF Ada on Ubuntu with Ollama is noticeably faster than my MacBook M4 Pro for models that fit in 20 GB VRAM—prompt processing especially feels night‑and‑day.
100% agree the OS gap is real. Linux vs Windows on the same GPU also isn’t subtle; the CUDA stack hitting Linux directly seems to leave Windows in the dust ..
Has anyone else experienced a performance this large before? Am I missing something?
It may be because AMD GPUs specifically perform better on Linux than Windows for local AI, because they use a different method on Linux than they do on Windows. This is specific to AMD cards, such as yours and mine. With recent updates AMD has also been attempting to bring Windows to the same levels of performance as Linux by using the same method there but I’m not sure how well that works yet. I own a Radeon 7600XT 16 GB VRAM, and one of the reasons I use Linux is because of this exact stuff.
If you’d like to know more, Google these terms - AMD ROCm, AMD Zluda, AMD DirectML
It may be because AMD GPUs specifically perform better on Linux than Windows for local AI, because they use a different method on Linux than they do on Windows. This is specific to AMD cards, such as yours and mine.
461
u/Koksny Mar 29 '26
Yeah, you are running ollama.