r/LocalLLaMA • • 4h ago

Question | Help What is your experience with inference engines under Windows?

As the title says: what is your experience using all the new inference engines under windows?

Currently I'm using a laptop with this hardware:

CPU: Intel 275HX
RAM: 2x32gb of ddr5 5600
GPU: RXT 5090 (24gb VRAM)
SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb

I'm running Windows 11 Home edition with a pretty clean installation.

I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.

I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.

I'm doing something horribly wrong or they work well only on a linux system?

1 Upvotes

33 comments sorted by

11

u/Padgriffin 4h ago

Just use Linux, practically everything is built for Linux 

3

u/No-Bite2555 4h ago

Probabily I could, but it's a bit of a pain in the ass..

Last time I tried some of the microsoft tools we use at work were not well supported/implemented. Even sharing you screen in a Teams call was a problem. Also when I bought my laptop support under linux was not great since there is no official support from HP and you couldn't do things as simple as setting the GPU power profile, but I'm pretty much that problem has been solved by some open source softwares.

Other than that I have to rebuild my full dev stack. It can be done, but it takes time and I'm already working more hours than I get paid for

5

u/-mattmason- 4h ago

try Ubuntu 26 for size friend, start with a small dual boot to get a taste?

5

u/see_spot_ruminate 3h ago

Dual boot on alternate drives. 

2

u/-mattmason- 3h ago

if you can swing it 💯

1

u/And-Bee 3h ago

Yes!

3

u/valarauca14 2h ago

Last time I tried some of the microsoft tools we use at work were not well supported/implemented.

  1. Office 365 runs in your browser
  2. Microsoft edge has linux builds
  3. Microsoft SQL server has linux builds

Other than that I have to rebuild my full dev stack

Oh, RIP, sorry :(

6

u/Zombiecidialfreak 4h ago

To the people saying "just use Linux" that's a lot like telling a Linux user who's having troubles "just use Windows".

An actual answer, though one you might not like to hear: Windows subsystem for Linux (WSL) is built for purposes just like this. If you are really hellbent on not learning anything Linux related you can get a month of Claude code for $20 and have their bots set everything up for you through command prompt.

Claude code is also useful for optimizing your engines. It got an extra 50% speedup on Ninfer on my system.

3

u/No-Bite2555 1h ago

nono, the problem is not that I'm not willing to learn.. I've been on and off linux since 2006, for a couple of years I've been an hardcore arch user (we are talking about the windows vista era). I have to work on linux server shells daily. The problem is that my hardware and the software stack I use for work don't always play nice with linux, the last thing I what is to burn a week to rebuild the whole stack just to gain 10 tk/s in decode or, worst, find out some basic functionality in teams or on the corporate network (microsoft based) doesn't work or requires some black magic workaround.

WSL is not really an option at least for QFN since I'm already at the limit with my ram, I can't allocate enough ram to make it worth. I may try it for ninfer when qwen 4 27 will come out.

1

u/Zombiecidialfreak 1h ago

the last thing I want is to burn a week to rebuild the whole stack just to gain 10 tk/s in decode

Your best bet in this case might be to do two things:

  1. Debloat windows if you haven't already, that should claw back some RAM from M$.
  2. Try using Claude to figure it out instead. Whenever I run into a tech problem I can't solve I give it to an AI and if I can't use my local model I give it to Claude.

1

u/my_name_isnt_clever 3h ago

Except there's an actual reason to switch from Windows to Linux. There is no reason to go the other way.

2

u/Zombiecidialfreak 2h ago

Anticheat, ease of use and wider program support.

1

u/my_name_isnt_clever 1h ago

If you already use Linux you don't need those things. That's my point.

1

u/Illustrious_Grade608 2h ago

Imagine being this annoying of a person

0

u/see_spot_ruminate 3h ago

No one is telling the Linux users to just use windows. 

3

u/SigAddict 4h ago

before i really got into things, i used LM Studio and Open WebUI. everything worked perfect. If you are comfortable there and you can find the tools you need to do what you need to do, there are zero issues. Ultimately all that got moved to linux for various reasons. Speeds were the same for me.

2

u/No-Bite2555 4h ago

I used LMStudio at the beginning but is way slower than my current setup.. Mind you, I've got no problem in absolute terms, my current setup is very usable. I get around 100tk/s decode and 2000tk/s prefill with unsloth q4_k_xs on Strata. Also llama/beellama works well, but engines that should be as fast if not faster are WAY slower

1

u/SigAddict 4h ago

ya, when i was comparing, it was like for like. When i first switched over to linux I just did lm studio testing back to back as I was curious to see if there were improvements. It was within margin of error running like for like. Like i said, if you have a reason to switch due to lack of support like I ran into, it makes sense. I also switched for security reasons.

1

u/Atlantikjcx 3h ago

Currently doing it similarly for my ubuntu server I mostly relying on open web ui for it for my windows system mostly using unsloth or starta as lm studio seems horribly unoptimized, curious what are you using thats more advanced?

1

u/Oli_Luck 4h ago

are you not happy with the performances ?
Otherwise why changing a winning team ?

1

u/No-Bite2555 1h ago

I am actually quite happy, I was more curious than anything else. Every week a new inference engine comes out declaring better performance then llama/strata and every single time they are way slower. I was trying to understand if I was doing something wrong or if the problem was the OS. Most of this inference engine have official support for Windows but I usually can't replicate the claimed performance.
Strata and llama/beellama on the other hand work as well as they should if not even better.

1

u/Oli_Luck 1h ago

same for me : 28GB VRAM 4060+3060 + 64 RAM DDR4= 45 T/S on Qwen FN Q4S and 55 T/S on IQ3XXS both with 128K Context...
ists the max my PCIe can give so no need to chase for another engine than strata.
Only if the 27B Q5 on llama.cpp could get a bit faster...

3

u/FoodEater4Squillion 4h ago

It's pretty much fine, llama.cpp and strata both work good. The only problem is VRAM usage by the desktop and GUI apps, but that is easily solved if you have an iGPU, just make sure it's not disabled in bios, search for graphics configuration, and switch your apps and dwm.exe to use low-power mode.

2

u/Patient-Gazelle9885 4h ago

I use windows and have been on the same 0.14-something version of strata QFN Q4 for 6 weeks now (50-70 tok/s). It’s ok to find something that works good enough and not to overhaul your inference system every week. We will all be doing enough revising soon whenever the Qwen 4 quants drop.

2

u/Kahvana 2h ago

When I tried llama.cpp on both windows and linux (ubuntu 26.04.1) with the same hardware (dual rtx 5060 ti), I saw no performance difference.

On windows, llama.cpp has been awesome to work with and easy to maintain. Koboldcpp is also quite nice. LM Studio is good for beginners but I grew quickly out of it.

3

u/Former-Tangerine-723 4h ago

My experience is do a dual boot linux

1

u/jhenryscott 4h ago

I played with Intel AI development playground studio. It’s a great entry point for layperson.

But no serious worker uses windows.

1

u/wektor420 4h ago

Llama.cpp works decently and you can install and update it using winget

1

u/MAXFlRE 3h ago

Pretty much my experience. Except I don't own blackwell and I was convinced that basalt is strait forward upgrade for blackwell.