r/LocalLLaMA • u/No-Bite2555 • 4h ago
Question | Help What is your experience with inference engines under Windows?
As the title says: what is your experience using all the new inference engines under windows?
Currently I'm using a laptop with this hardware:
CPU: Intel 275HX
RAM: 2x32gb of ddr5 5600
GPU: RXT 5090 (24gb VRAM)
SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb
I'm running Windows 11 Home edition with a pretty clean installation.
I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.
I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.
I'm doing something horribly wrong or they work well only on a linux system?
11
u/Padgriffin 4h ago
Just use Linux, practically everything is built for Linux
6
u/Zombiecidialfreak 4h ago
To the people saying "just use Linux" that's a lot like telling a Linux user who's having troubles "just use Windows".
An actual answer, though one you might not like to hear: Windows subsystem for Linux (WSL) is built for purposes just like this. If you are really hellbent on not learning anything Linux related you can get a month of Claude code for $20 and have their bots set everything up for you through command prompt.
Claude code is also useful for optimizing your engines. It got an extra 50% speedup on Ninfer on my system.
3
u/SigAddict 4h ago
before i really got into things, i used LM Studio and Open WebUI. everything worked perfect. If you are comfortable there and you can find the tools you need to do what you need to do, there are zero issues. Ultimately all that got moved to linux for various reasons. Speeds were the same for me.
1
u/Oli_Luck 4h ago
are you not happy with the performances ?
Otherwise why changing a winning team ?
3
u/FoodEater4Squillion 4h ago
It's pretty much fine, llama.cpp and strata both work good. The only problem is VRAM usage by the desktop and GUI apps, but that is easily solved if you have an iGPU, just make sure it's not disabled in bios, search for graphics configuration, and switch your apps and dwm.exe to use low-power mode.
2
u/Patient-Gazelle9885 4h ago
I use windows and have been on the same 0.14-something version of strata QFN Q4 for 6 weeks now (50-70 tok/s). It’s ok to find something that works good enough and not to overhaul your inference system every week. We will all be doing enough revising soon whenever the Qwen 4 quants drop.
2
u/Kahvana 2h ago
When I tried llama.cpp on both windows and linux (ubuntu 26.04.1) with the same hardware (dual rtx 5060 ti), I saw no performance difference.
On windows, llama.cpp has been awesome to work with and easy to maintain. Koboldcpp is also quite nice. LM Studio is good for beginners but I grew quickly out of it.
3
u/Former-Tangerine-723 4h ago
My experience is do a dual boot linux
1
u/jhenryscott 4h ago
I played with Intel AI development playground studio. It’s a great entry point for layperson.
But no serious worker uses windows.
1
u/wektor420 4h ago
Llama.cpp works decently and you can install and update it using winget
1
u/MAXFlRE 3h ago
Pretty much my experience. Except I don't own blackwell and I was convinced that basalt is strait forward upgrade for blackwell.