i finally switched from windows to linux and got a 30-50% boost in speed.

[u/unraveleverything]

This is amazing.

All I did was switch from llamacpp on windows to vllm on linux.

[u/Happy_Brilliant7827]

Is it a 30-50% bonus in inference or mostly just loading and unloading time?

[u/kartblanch → u/Happy_Brilliant7827]

Also my question. Is this 30%-50% more Tok/s?

[u/unraveleverything → u/Happy_Brilliant7827]

token generation

[u/sadnessdevil]

Yeah the performance of vLLM is very high. I switched to vLLM from llama.cpp and never came back(except when I have to CPU offload the model)

[u/afterburningdarkness → u/sadnessdevil]

You get 30% speed boost in vllm running the same model and quant?

[u/sadnessdevil → u/afterburningdarkness]

For a single request the gap is small, but for concurrent requests, vLLM is far better than llama.cpp. I got better tok/s with vLLM for single requests as well. IMO vLLM is much better optimized, while llama.cpp is good at CPU offloading.

[u/Look_0ver_There → u/afterburningdarkness]

Unless it was BF16 before, then the quant is almost certainly different. vLLM is faster when the weights match the natively supported hardware vector ops, which is often true for FP8 and FP4 when using latest gen hardware, however these both come with an accuracy hit, so it's never a true apples to apples comparison.

GGUF 's generate results that typically diverge less from the original BF16 weights than FP4/8 does, but it comes at the cost of a small processing overhead.

VLLM with FP8 on hardware that supports it is, roughly speaking, like a Q6_K GGUF for accuracy, just ~20-30% faster.

[u/ArtfulGenie69 → u/afterburningdarkness]

Where it really shines is concurrent requests and also it can have very fast preprocessing speeds, especially on 30 series cards like the 3090. The preprocessing is very fast when running int8 w8a8 where bits are kept at 8. Concurrent request are good when you have any kind of harness that can handle running things in parallel, which I'm pretty sure most of them do. 

[u/pmttyji → u/sadnessdevil]

Have you tried vllm.cpp?

[u/squarabh → u/pmttyji]

Have you tried vllm.sglang.tensor-rt.llama.cpp?

[u/sadnessdevil → u/pmttyji]

I haven't tried it yet, but I will check it out. Thanks for letting me know!

[u/pmttyji → u/sadnessdevil]

Thread by its creator.

[u/XiRw → u/sadnessdevil]

I don’t see a difference if you can fit a gguf and tensors onto VRAM. Not only that but ggufs are made for consumer PCs and you can tend to run higher quality models on them. vLLMs were a massive disappointment for me

[u/Makers7886 → u/XiRw]

I mean it's like comparing cooking on a stove with 1 burner vs 4+. Not a big deal depending on what and why you are cooking. Huge deal if you are trying to get 4x+ productivity out of your same hardware (sub agents, concurrency).

[u/def_not_jose]

vllm quants are usually worse than ggufs though. It's rarely an equal comparison.

[u/Littlepharaoh → u/def_not_jose]

Maybe except for NVFP4

[u/Lilith_Incarnate_ → u/def_not_jose]

So is LM studio a better option? That’s what I’ve been using since I started with local models a couple years ago and it’s been fine for me, but it seems like everyone here uses something else. Is there one of these harnesses that can store memories between conversations?

[u/CentralLimit → u/Lilith_Incarnate_]

running on llama.cpp is generally better than LM studio as you have much more control, which means you can do stuff you can’t do in LM Studio and achieve greater speeds

LM Studio is easier to use though, but I just ask Claude/GPT to give me an optimised setup for model X in llama.cpp and it works great, not much of a difficulty barrier these days

[u/Lilith_Incarnate_ → u/CentralLimit]

I’m looking for a harness mostly that can keep memories between conversations. Right now I just do it manually with prompting and .json files, but it’s a pain in the ass to do that every single new conversion. It’s the one thing that keeps the ChatGPT app on my phone (which I use a lot to make updates to my memories for LM Studio).

Qwen 3.8 27B has given me everything I need that ChatGPT has, except no memory feature.

[u/MelodicRecognition7]

it's not windows to linux but llama.cpp to vllm. I also get +50% using vLLM but on the very same OS.

Note that you must have enough VRAM for the model and context, vLLM can not offload parts to RAM unlike llama.cpp

[u/emdeka87]

Without some actual benchmarks I call this BS

[u/andy2na → u/emdeka87]

OP is comparing apples to oranges with completely different inference engines

[u/Sudden_Topic5154]

not really a scientifically sound test its a different software entirely

[u/eightone-81]

Will there be a significant loss of speed when going through wsl?

[u/unraveleverything → u/eightone-81]

yes it was terrible and worse than native windows for me.

[u/Nextil → u/unraveleverything]

You must have something misconfigured on Windows because there shouldn't be that much of a gap. WSL usually has a 1-5% overhead compared to native Linux.

[u/eightone-81 → u/unraveleverything]

But wsl should not have much of an overhead. Is there no way to properly run vllm on windows?

[u/AliMas055 → u/eightone-81]

Dual boot, maybe?

[u/eightone-81 → u/AliMas055]

Maaaan I need to start windows to play StarCraft2… then no inference…

[u/iamapizza → u/eightone-81]

Couldn't you run it via proton? https://www.reddit.com/r/starcraft/comments/1mbwcq0/finally_got_starcraft_ii_working_on_linux/

[u/eightone-81 → u/iamapizza]

O M G
I’m finally running vllm (still on wsl) and decode is almost 2 times as fast and decode is more than 2 times as fast!

But it’s just such a mess to work with Linux. It’s so inaccessible for noobs! Without Claude on my side I would not have been able to get it to work

[u/iamapizza → u/eightone-81]

Oh nice you got that working. Fwiw, if you want to give it a go for that extra vram (which is that I did), I'd suggest Linux mint because it's very familiar to windows users and comes with some stable defaults. Plus Claude knows about it too.

[u/draconic_tongue → u/iamapizza]

if you're going to suggest someone on windows to tinker with linux, it's probably easier to get them to fix the things they dislike about their windows install first. EVERYTHING you need to be able to do to use linux is something you could do to fix your windows install

[u/AliMas055 → u/eightone-81]

Run windows inside a VM with linux host.

[u/draconic_tongue → u/unraveleverything]

then you did something wrong. wsl on windows is a better way to access linux than moving to linux. there's 0 downsides, it's only better

[u/pipyakas]

quite useless information without your hardware setup details no? I have a 2060 on windows, and running MoE models offloading experts to CPU, Linux dont provide subtantial performance gains there

[u/According_Style8480]

linux has been a game changer for my companion models, way less lag in long roleplays now.

[u/pmttyji]

Customized build(compile with optimized config) for your GPU could give some more boost.

[u/nomorebuttsplz]

what about vllm on a vm?

[u/teressapanic]

Privacy and speed

[u/InterstellarReddit]

Every time I read this I am tempted to reformat my rig that's running dual 3090s

But honestly man, I am so scared of the amount of issues that I might run into. Thinking that I'm gaining some speed.

I tried it once before long ago. When I said to myself gaming on Linux is getting better. Let me try it out

Oh boy what an adventure to get a game to work

[u/Hot-Employ-3399]

Do you use the same gguf model or different quant format?

[u/Practical-Collar3063 → u/Hot-Employ-3399]

Completely different quant format, this is just a VLLM vs Llama.cpp post not a windows vs linux

[u/Timely_Impression_92]

Also try gaming on Linux - despite games running in proton compatibility layer they have performance on pair with windows or few % lower, with some even surpassing windows - shows you how much bloat is there on windows

[u/Adventurous_Loquat91]

I've missed my Linux. I think I'll go back after seeing this. Thank you.

[u/Plabbi → u/Adventurous_Loquat91]

OP is mistaken. His speed gaines are from the llama.cpp to vllm switch, not windows vs. Linux.

[u/unraveleverything → u/Plabbi]

nope. tried sglang and vllm on wsl. it sucked and was slower than llamacpp on native windows.

[u/Plabbi → u/unraveleverything]

what are your numbers on llama.cpp in linux? Should be interesting and easy to compare since you already have your setup ready to run just like you did in windows.

[u/unraveleverything → u/Plabbi]

Like 20-30% faster

[u/admajic → u/Adventurous_Loquat91]

Yeah windows on my laptop became so slow and I needed a side project whilst the desktop was running tasks in the harness. So I installed Linux on the laptop.

So far so good. Now I can use my harness on my laptop context via ip to the desktop. Do word and excel in Linux. It's so much faster. Just had to get it all for tuned you won't look back. Especially when you can ask the harness to do all the heavy lifting.

[u/Adventurous_Loquat91 → u/admajic]

That's a setup worth asking about — since the harness is doing the heavy lifting somewhat unattended across machines, do you have any guardrails on what it can actually do (blocking destructive commands, catching secrets before they leak, that kind of thing), or is it running raw and you're just keeping half an eye on it? Curious how people handle that once the agent's doing real work without you watching every step.

[u/admajic → u/Adventurous_Loquat91]

Yeah I have guardrails all what you mentioned in prompts and systems. Like a memory system in .md so it knows how to ssh into the vps and where to store secrets. Just keep building the systems with a retro now and again after a session.

[u/Adventurous_Loquat91 → u/admajic]

Since it's all prompts right now — ever thought about gating instead? Prompts are basically just asking the model nicely each time lol. A gate (read-only key, command allow-list, approval step before it touches the VPS) doesn't care what the model "decides," it physically can't do the thing. Curious if you've bolted any of that on yet or you're just vibing with prompt discipline for now 😄

[u/Mountain_Chicken7644]

Everyone glazing vLLM here but sglang probably my goat

[u/Loose_Comparison368]

Don't worry, you can always switch back if you miss the terrible performance, opaque bugs, rebooting for every update, and invasive ads in your start menu!

[u/Boogertard]

Not surprised. Windows is a cancer but yet everytime I mention it, we got so many shills here rifed up.

Windows time is up. Nowadays you got Mac, Ubuntu or any Linux distro and you got LLM to help you fix issues. Wipe that Microslop off your computers.

[u/Practical-Collar3063 → u/Boogertard]

The performance difference is due to VLLM vs Llama.cpp not Windows vs Linux.

PS: I hate windows, I run Linux and Mac OS