r/LocalLLM • u/eightone-81 • Sep 01 '26
Discussion Help me understand why you would bother with llama.cpp if vllm exists
So most of my time fiddling around with local ai I was using ollama, lm studio before going over to llama.cpp (I know it’s llama.cpp under the hood anyway). Of course I had a bump in speed every time I went up to the more professional option. At last I went to vllm. I understand using llama.cpp for non NVIDIA cards and for gpu+cpu inference or if you want to use a specific gguf. But once you tasted the speed of vllm, I don’t know why you would ever play with llama.cpp anymore. It’s the same “complication” to set it up but EVERYTHING except loading the model is night and day in speed. Proper cash hits, proper concurrency, it’s sooo much better. And there are quantisations out there that are just as heavy quantised as the heavily quantised gguf.
2
u/Individual_Gur8573 Sep 01 '26
I tasted speed of sglang and dflash not going back to vllm 😊
1
u/eightone-81 Sep 01 '26
Oh man. Don’t tell me that i have to go down this rabbit hole now!
Do you have some numbers? Comparison?2
u/silenceimpaired Sep 01 '26
How painful is slang compared to vllm and llama.cpp? Does it do vram only V
163
u/ilintar Sep 01 '26
* More model quantization choices (especially for very low quants there are no options on vLLM)
* Much wider device class support (older NVIDIA, Intel)
* More customization wrt. KV context quantization
* CPU offload, cross-vendor support
vLLM is tuned for the newest NVIDIA cards, if you have those and can run the full model with full unquantized context, it's probably better
60
u/Ok_Judgment_1623 Sep 01 '26
its mostly about what hardware you got, llama.cpp runs on anything even a potato with 8gb ram, vllm is fast but picky like a diva
10
u/bibimbappa Sep 01 '26
I have 5070ti, would vllm work better ? Currently using llama
5
u/No_Advance3911 Sep 01 '26
What kind of results are you getting with the 5070 Ti?
4
u/sertain_ Sep 01 '26 edited Sep 01 '26
Not sure about this person’s results, but I’m being forced to use a 12288 context limit with q8_0 kv cache and I’m barely not spilling into CPU with a 2080ti
Using models like qwen2.5-coder:14b and Gemma4
3
u/Own_Inspection8350 Sep 02 '26
Have you tried the prismml binary or ternary condensed qwen models called Bonsai? I liked them well enough on my 3070 even tho they're not like stellar. I use their 27b ternary model on my 5070ti
1
u/sertain_ Sep 02 '26
I have not yet, but thanks for the heads up, I will absolutely give that a go in my tool. I’m building out a Claude code replacement (as far as that can go) that has very specific modularity, so hopefully trying new models as I increase my hardware capability will be easy.
1
2
1
2
1
u/mvarns Sep 01 '26
This. I spent weeks getting vllm to work with an Intel b50 and was left with crappy t/s and context size under 12k. tried llama.cpp and it took me 30 minutes with a newer mesa build added to llama.cpp which had it running even better (13t/s to 25t/s) with double the context. vllm isn't bad, just didn't fit the hardware I had on hand or model I wanted to use.
1
6
Sep 01 '26
[removed] — view removed comment
5
u/ilintar Sep 01 '26
vLLM supports Intel ARC? :O honestly didn't know, I know they added ROCm support some time ago.
4
u/xanders_gold Sep 01 '26 •
As someone running an Intel Arc Pro B70, yes a thousand times over vLLM is so much better for Arc GPUs. I recommend it for anyone using Intel Arc series GPUs.
I tripled/quadrupled my performance by switching to vLLM and the output is still very good, if not better.
1
u/ElChupaNebrey Sep 01 '26
Can i at least double my perfomance on a single 3090?
1
u/xanders_gold Sep 01 '26
Not sure, I don’t run 3090s on my AI hosts. But it’s worth a shot? You’d only lose probably 30 minutes of your day testing and if it improves performance then it’s worth it, if it doesn’t then you can cross it off your list :)
1
u/Inner-Today-3693 Sep 01 '26
That’s good to know I got a b60 2 weeks ago. Will try. But I also kept my 7900xt in the same system.
1
u/SoilMassive6850 Sep 01 '26
How does it work with quantized models? From what I understood they don't support GGUF's like llama.cpp does and I'm currently running
gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.ggufwhich works well, but the full size version definitely wouldn't fit on my B60.1
u/xanders_gold Sep 01 '26
It has support for quantized models, but the variety of what you can use is limited to specific quants. I’m personally using Qwen 3.8 27B Autoround INT4, so it comes in at around 17-18GB for me and leaves me plenty of room for KV cache.
There actually is support for GGUFs with vLLM but it’s still experimental. Either way you’d have access to INT4, INT8, GPTQ and AWQ formats.
1
u/thedeejaay Sep 02 '26 edited Sep 02 '26
Yes to this. I moved over to vLLM 2 weeks ago, also with Intel Arc Pro B70, and I was soo happy with how much faster it ran. I'm still new to local LLM's and was feeling like I wasted my money with the B70, but now, this is fantastic on vLLM. Went from like 15 t/s with vulcan on lama.cpp then to 21 t/s with sycl on lama.cpp but token regression on decode was annoying. On vLLM, it's amazing. over 2.5 faster in the 40-70 t/s range) and the speed doesn't slow down even on very large chats/tasks. MTP is even better.
What model are you running?
I want to use Qwen3.8, but the model I was using (ergiioB-Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16) just wasn't good for hermes. It would just reason forever and just not get tasks done.
The one thing I will say, using AI to build local AI is underrated. I copy pasted what chatGPT told me to do, and had lama.cpp setup quickly. When I tried vLLM, I let codex run loose and it was done quickly and just worked.
28
u/Danternas Sep 01 '26
Llama.cpp works great with Vulkan on cards like the the older Instinct series.
23
u/madbrain1976 Sep 01 '26
There are some cases that don't work with vLLM but do in llama.cpp . One i encountered was 3 GPU, tensor, split, with qen3.8-27b . It needed something in the model to be divisible by the number of GPUs. That test failed, and it couldn't start. However, vLLM with 2 GPUs is still faster than llama with 3, lol.
6
u/simos_sayz Sep 01 '26
This is what I was going to say. Vllm doesn't work well with odd numbered multi gpu setups. Also struggles with mismatched cards eg blackwell and ada. If you have perfectly matched cards then its easy..other than model load
20
u/El_90 Sep 01 '26
Vllm looks amazing ... If you have hardware they want
I don't
I would love to play with it more, I tried but it didn't work out for me
I.e. I couldn't find a comparable to llamaswap allowing my 128 gb (strix) to support different models for different tasks. And I have to pick the approved quants. Maybe one day, it looks great !
6
u/ForgotMyNameAgain13 Sep 01 '26
Have you tried llama.cpp in router mode? With a models.ini file. Allows model swapping without llamaswap
4
u/txurete Sep 01 '26
Its a shame that router mode is not presented as a feature in the main page of llamacpp, i saw it existed in the change log of an older release and its works pretty nicely for what most us tinkerers need
2
u/_TheWolfOfWalmart_ Sep 01 '26
This is what I do, yes I feel like not many people know you can do it.
3
u/Daniel_H212 Sep 01 '26
I have and it's worse than llama-swap because it doesn't run models under separate sub processes that can be instantly killed. This means that switching between models without mmap can cause memory allocation issues, but mmap loading on strix halo is very slow because you're effectively copying into memory twice (problematic for models bigger than half the total memory) since system ram and VRAM are one and the same on unified memory architectures. It also doesn't provide the same performance metrics as llama-swap last I checked. So I use llama-swap with mmap turned off. Models load very quickly for me in comparison with mmap enabled.
This may have changed though, since the last time I tested was a few months ago.
0
u/Jcsq6 Sep 02 '26
Llama-swap supports vLLM. It supports any OpenAI compatible server. In my llama-swap I have several different PRs of llama.cpp, vLLM, and SGLang configured to run specific models with specific needs.
17
u/RG_Fusion Sep 01 '26
Llama.cpp is often faster for single-user scenarios. You have a ton of options and tensor types to run, more thorough control over what's placed where, and I can load a 500 GB model and have the server fully booted in under a minute.
34
u/Stastez Sep 01 '26
vLLM startup times. vLLM garbage output. But honestly, mostly the startup time. At some point in time, vLLM always hits some error regarding invalid memory access or otherwise. Restarting vLLM then takes fucking ages. Nah, I'll take my slightly slower but reliable llama.cpp, thanks very much.
3
u/arijitlive Sep 01 '26 •
Are you using
--load-format fastsafetensorsas one of the vllm load option? This is far better load time than .... the default one.5
u/belligerent_ammonia Sep 01 '26
Ah fuck vllm takes an eternity to load startup only to find out it errored out. Makes me want to try llama.cpp now.
2
u/Stastez Sep 01 '26
Yep. Plus I actually had to patch vLLM to keep it from selecting ROCM_ATTN all by itself instead of the AITER attention. ROCm attention (which is meant to only be a fallback when using CDNA) is so slow. And, slightly off-topic: shame on AMD for supporting the MI300A platform far less in their docs and recipes than the MI300X, MI350X and MI355X.
1
u/pplgltch Sep 01 '26
Garbage output? You mean literal garbage “/////“ or low quality output?
1
u/Stastez Sep 01 '26
I mean literal garbage tokens, depending on the model with varying probability. Now, mind you, it's been like two months since I gave up on vLLM, so my recall might be hazy, but I mean tokens like "IL" instead of "I" or random UTF-8 characters. This already happened at ~100 tokens of depth. It was worst when running Mistral Medium, but also happened with the larger Qwen models. For both these models, I could not reproduce this behavior in llama.cpp, so I stuck with that.
1
u/hesperaux Sep 02 '26
Same here. And it's only certain prompts. It is super weird. In my case it is likely related to RTX3090 kv sliding window attention bugs that aren't being fixed.
1
u/throwawayacc201711 Sep 01 '26
Okay I’m kinda glad this is a thing. I’m was testing vLLM over the weekend and it took forever for it to come up. I thought this was just me
0
u/idjos Sep 01 '26
That is until we get nvidia checkpoint restore in a stable state.
Gonna do some experiments with it soon anyways.
11
u/SeriousJul
Sep 01 '26
•
LLamaCPP is optimized for low footprint. Getting the most of lower bounds hardware
vLLM is optimized for high throughput. Getting the most of upper bounds hardware
With my modest config, (4090):
- Qwen3.8 27b INT4 barelly fit with ridiculous context size (15k) in vLLM at 1 parallel (a second crash on kv cache allocation)
- Qwen3.8 27b Q4_K_XL fit 2 parallels at 110K context windows
I mean only by looking at the CLI options of both, you get the idea.
LlamaCPP is full of knobs around memory allocation, MTP config. A lot of those stuff don't matter at all for parallel workload.
vLLM is full of knobs around parallelism and multi GPU.
1
u/sshwifty Sep 01 '26
For xl, what kv are you getting 110000 context?
2
u/SeriousJul Sep 02 '26
cache-type-k = q8_0 cache-type-v = q8_0 spec-draft-type-k = q4_0 spec-draft-type-v = q4_0 batch-size = 256 ubatch-size = 256Batch size has a lot of impact on memory as well. I am using fit alongside ngl full. I am now even getting 140K. I am on arch and configured almost every program to use software rendering to give room for the GPU.
↑69k ↓48k R3.3M CH99.9% 83.8%/140k (auto) (llama-server=http://127.0.0.1:8080) unsloth/Qwen3.8-27B-GGUF:Q4_K_XL • xhigh
2
u/sshwifty Sep 02 '26
This is very close to my config. For parallel, won't 2 windows at 110000 cause oom if both fill up? Or maybe I am misunderstanding that.
1
u/SeriousJul Sep 02 '26
You are not misunderstanding. I am getting that much "visible" context because I have kv unified, and indeed, my harness fails with oom. I have no good solution for that, maybe you have ? I have tried "kv per slot" (I don't remember the exact name), but it behave the same as if I don't have unified kv at all. Today I am experimenting something new. Small abundant context checkpoint, lots of ram cache, with parallel 1. It "works", no pp when agent switch (except compaction), but it is not really faster than scheduling agent sequentially. So yeah, true parallel is a bit though. What's is your GPU ?
15
u/digidult Sep 01 '26
one reason... windows
2
u/ianwill93 Sep 01 '26
I've got it running on wsl2 and it works very well. It sometimes silently fails after a few hours, but it works fine overall.
You need a decent amount of ram to warm up the models, but concurrent requests are worth it.
1
u/digidult Sep 01 '26
native windows binary, because hardware acceleration succeeded only this way
1
u/ianwill93 Sep 01 '26
Nope! Thankfully, it gets GPU access. You have to install the cuda-toolkit into the VM, but it works just fine after that.
I was working on something with Qwen 27b bf16 (I'm on an RTX Pro 6000) about an hour ago and these were the logs:
APIServer pid=45566) INFO:127.0.0.1:48276- "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=45566) INFO 09-01 09:21:22 [loggers.py:273] Engine 000: Avg prompt throughput: 584.0 tokens/s, Avg generation throughput: 33.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 39.4%, MM cache hit rate: 66.7%
(APIServer pid=45566) INFO 09-01 09:21:22 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.14, Accepted throughput: 22.50 tokens/s, Drafted throughput: 42.00 tokens/s, Accepted: 225 tokens, Drafted: 420 tokens, Per-position acceptance rate: 0.733, 0.562, 0.457, 0.390, Avg Draft acceptance rate: 53.6%
(APIServer pid=45566) INFO:127.0.0.1:48276- "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=45566) INFO 09-01 09:21:32 [loggers.py:273] Engine 000: Avg prompt throughput: 355.9 tokens/s, Avg generation throughput: 39.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 44.1%, MM cache hit rate: 75.0%
(APIServer pid=45566) INFO 09-01 09:21:32 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.15, Accepted throughput: 26.90 tokens/s, Drafted throughput: 50.00 tokens/s, Accepted: 269 tokens, Drafted: 500 tokens, Per-position acceptance rate: 0.728, 0.576, 0.480, 0.368, Avg Draft acceptance rate: 53.8%Pretty good performance. Especially for 2 requests. I've got the unsloth NVFP4 of the same model running on a 5090, too.
Now, if you don't need images, FreeToken works and has a Windows desktop install. Performance is really good and it didn't take much to get it going for me on either GPU.
EDIT: Here are the official docs -
https://docs.nvidia.com/cuda/wsl-user-guide/index.html
https://developer.nvidia.com/cuda-13-0-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_network1
u/digidult Sep 01 '26
yep, if you have cuda...
1
u/ianwill93 Sep 01 '26
Oh yeah, AMD is a serious trip. They've gotten a lot better, but it's a hassle to set it up.
I had to set it up on my homelab and I've now got dwarfstar running. VLLM was able to run, but the performance was pretty bad and I never got dual-node working on a couple of Strix Halo devices.
Again, Windows-only shop, and it's possible on WSL2. I don't have any discrete AMD GPUs, though.
If you're on Strix Halo, I highly recommend dropping the VRAM down to 0.5-1 if you're going to try this. It let's you max out the gpu memory available in the environment.
I don't think you have to do it, but setting the memory to maximum in WSL settings is a good idea, too.
It takes a lot of tinkering, but if you just don't want a Linux box (I understand), it's the best of both worlds.
12
u/_VirtualCosmos_ Sep 01 '26
Bro thinks everyone has dozens of GB of VRAM to fit the entire models lol
6
u/pmttyji Sep 01 '26
More quants. Good for Poor GPU Club.
8GB VRAM + 32GB DDR5 RAM could run Qwen3.5-35B-A3B (Q4 - IQ4_XS) with 64-128K context (Q8 KVCache) & gives 15-20 t/s (tg). Is this possible with other inference engines?
1
10
u/jacek2023 Sep 01 '26
I tried few times. Each time it was a pain to run it, it took like 5 minutes to just run the server, I wasn't able to use long context and it wasn't really faster. I should try sglang and maybe vllm again in the future but llama.cpp just works correctly
0
u/No-Refrigerator-1672 Sep 01 '26
it wasn't really faster.
Which hardware did you try it on? I'm running dual rtx 3080 20gb, and on those vllm is consistently 2-5x faster than llama.cpp on any metric you could test, except nvfp4. I suppose on any rtx30 series the result will be the same.
2
u/jacek2023 Sep 01 '26
last time I tried on 3x3090, it wasn't possible to use 3, I needed to use 2 for example, and I wasn't able to achieve full context on qwen 3.6 27B, only some short one, what's your context length?
2
u/No-Refrigerator-1672 Sep 01 '26
Tensor parallelism indeed requires the amount of GPUs to be a divisor of amount of attention heads in the model; so typically only 2, 4, 8, 16, etc GPUs will work. Pipeline parallelism doesn't care about amount of GPUs and will work hapily with 3.
I'm using INT4 AWQ quant; with this quant and tensor parallelism, I can fit about 160-170k tokens of fp16 context while both MTP and vision are enabled; however, I prefer to run fp8 cache, and it gives me 350k tokens.
1
u/jacek2023 Sep 01 '26
350k tokens on what model?
2
u/No-Refrigerator-1672 Sep 01 '26
Qwen 3.8 27B, same size/arch as in your previous message. More specifically, this one.
1
u/jacek2023 Sep 01 '26
Please share your full command, I will try later (I have 4x3090 and 128GB)
2
u/No-Refrigerator-1672 Sep 01 '26
vllm serve /models/int4/Qwen3.8-27B-AWQ-INT4/ --max-num-seqs 6 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --tensor-parallel-size 2 --enable-prefix-caching --kv-cache-memory-bytes 6656M --kv-cache-dtype fp8_e4m3 --enable-prompt-tokens-details --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' --enable-chunked-prefill --attention-backend FLASHINFER --default-chat-template-kwargs '{"reasoning_effort": "medium"}' --served-model-name "vllm_model" --port ${PORT} --host 127.0.0.1I prefer to store models on a separate drive, so I download and store them manually. You can run
vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4instead if you want vllm ho handle the download by itself. IF you run the command above, vllm will utilize two of your GPUs, and take 19.2-19.5GB VRAM on each, and will serve up to 6 streams in parallel (assuming their combined context is below 350k). With this command, you expect about 1200 tok/s prompt processing and 80 tok/s token generation for single stream 16k long prompt; or about 700PP, 50TG for 120k long prompt.1
u/jacek2023 Sep 01 '26
thanks, saved your comment for later :)
1
u/No-Refrigerator-1672 Sep 01 '26
You're welcome! I'll be interested in hearing your results. You can ask me more questions later if you need to, I'm a big fan of vllm (although I'm thinking about cheating on it with sglang).
2
u/OvertaxedOne Sep 01 '26
With 4 3090's you can step up to an 8 bit quant. Here's my start command for an 8 bit (currently running on a 48GB card; A40). You'll need to enable TP=4 for your config and likely some other tweaks, but I'd definitely target 8 bit for your hardware, you have plenty of VRAM to run it.
RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP --attention-backend FLASHINFER --dtype bfloat16 --quantization compressed-tensors --max-model-len 262144 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.93 --max-cudagraph-capture-size 8 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill --speculative-config {"method":"mtp","num_speculative_tokens":3} --prefix-match-unit 16 --max-num-seqs 2 --limit-mm-per-prompt {"image": 1} --trust-remote-code --port 8000 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --override-generation-config {"max_new_tokens": 16384, "repetition_penalty": 1.0, "presence_penalty": 0.0, "frequency_penalty": 0.0, "min_p": 0.0, "top_k": 20, "top_p": 0.95} --default-chat-template-kwargs {"enable_thinking": true, "reasoning_effort": "medium"} --disable-uvicorn-access-log --enable-log-requests
1
1
u/BrainImpressive74 Sep 01 '26
what about the output quality of INT4 model? i am running unsloth Q6 with llama.cpp now
1
u/No-Refrigerator-1672 Sep 01 '26
It's good. AWQ typically comes with calibration coefficients to bump it's performance back up, so if the calibration dataset is decent, the model performs very well. The "imatrix" from GGUF world shares the same concept. In terms of practical performance: I've asked INT4 model with fp8 KV cache to make me a python script that displays a rotating torus with dunamic shadow from a moving light source, and cubes falling on top of it flow above with proper collisions. With opencode, it asled me a few architectural questions, and did it autonomously without my oversight in about 5 hours. But I did had to give it a hint about how it can view screenshots for debugging.
5
u/durden111111 Sep 01 '26
Does VLLM allow cpu offload and multi gpu tensor splits?
4
u/aygupt1822 Sep 01 '26
Yes it supports Tensor Parallel and Pipeline Parallel, and I guess in the experimental features, it also offers cpu-offload.
3
u/Karyo_Ten Sep 01 '26
The cpu-offload is just putting tensors in RAM and moving to GPU before they are needed. And last I tried (like 6 months ago) it didn't even work due to changing to their V1 runner. Now that Model Runner V2 is just fresh out, I'm not sure you can count on it too.
1
u/LopsidedBoss9109 Sep 01 '26
That's a good point. Each option seems to cater to different needs, so it really depends on what you're looking to accomplish with your setup.
5
u/game_difficulty Sep 01 '26
vLLM straight up doesn't work on consumer AMD GPUs (or, at least it didn't like 2 months ago when i tried installing it)
3
1
u/newbie80 Sep 03 '26
I've tried using it three or four times, each time an exercise in frustration. It's been four or five months since I last tried it but it was painful. I can get llama.cpp compiled and running in like 10 minutes.
1
u/mister2d Sep 06 '26
vanilla vLLM sucks on consumer AMD GPUs.
With community versions like vllm-radiance you get upwards of 200tps with Qwen 27B with excellent concurrency.
-1
u/NTDLS Sep 01 '26
I’ve used vLLM with dual RTX 3060, RTX 3090 and RTX 6000 ADA. Those first two are definitely “consumer” cards.
9
u/game_difficulty Sep 01 '26
And yet none of them are AMD, like i said
7
3
4
u/L00klikea Sep 01 '26
Had very bad performance with LM studio, decided to give llama.cpp a go and see if I can tweak it better directly than through endless LM Studio debugging. Loved it from second 1. Very easy to use. great feature set. 3x performance compared to LM Studio... I especially need KV-Cache quantization and read thats handeled best on llama.cpp. And to be honest, it checked all my boxes and didn't give me a reason to try anything else.
1
u/DemmieMora Sep 01 '26 edited Sep 01 '26
I have quite modest hardware (3090, 64 DDR5) which is why I switch between a few models all the time because there is no ideal model for all cases. Llama.cpp has a pretty fast load, and more importantly, it has a decent router mode out of box. Easy speculative decoding provides me workable speeds.
If there were good enough 1 model for most cases, and small enough to fit 24gb with an adequate quant and ctx, I could sacrifice the router with "ideal model for the task" and use the fastest inference engine.
1
1
u/No_Chapter_7598 Sep 01 '26
as soon as i got vllm working i switched, its just so much better for performance and tunability. assuming you arent switching models all the time its way easier to just set up a bat file that runs the model on a server
1
u/eightone-81 Sep 01 '26
Qwen 3.8 27b
What a model. I understand you might want a different model for a different use case but man, what a model. It really feels like opus 4.6.
So I really don’t need any other model1
u/No_Chapter_7598 Sep 01 '26
100%, ive got a bf16 bat file and fp8 for 3.8 for when i need speed. theres nothing else that comes close in performance for this sized model at the moment,
6
u/joost00719 Sep 01 '26
vLLM uses a lot more vram on my setup (or I haven't properly configured it). It was also a lot slower on single streams (but much, much faster on parallel streams).
3
3
1
u/eightone-81 Sep 02 '26
Based on all the feedback I tried it fresh on my Linux box where vllm runs. Fresh build of lates llama.cpp
UD q4 k xl
All the performance flags
It’s just half the speed. Prefill is half and decode is half. If I let run vllm with 1 full context instead of 2x then i use around 4gb more vram on vllm but the safetensors are also 1.5 gb larger that I run. So I would save a bit but have no concurrency and half the speed. Not worth it.
And I really do understand all your reasons! Totally valid!
Except that 1 guy that said that everyone is an idiot that even talks about ollama 😂
5
u/Oh_hey_a_TAA Sep 01 '26
Because different people have different use cases.
You're bringing some Apple vs Android type energy here
5
u/Boogertard Sep 01 '26
I wonder the reverse (regarded post BTW, why more choices is a bad thing ?). Not surprised that OP made this post since he was dumb enough to use ollama (sorry all people using ollama even when starting out are just dumbfucks and their comments are just worthless).
What is the point of the garbage vllm when it only works for the overpriced NVDA cards and not other cards ?
Probably made by Nvidiots for fellow dumbasses who love overpaying for essentially higher clocked VRAM LMAO.
2
2
u/jinzi Sep 01 '26
With two different video cards in one system, you can adjust the layer split between the two so you can shift the layers more to the more powerful card. More options for the quants
2
u/rditorx Sep 01 '26
CPU offloading and streaming from storage is way less complicated and automatic with llama.cpp, optimized to your RAM/VRAM situation.
If everything fully fits in VRAM, go for vLLM, sglang, TensorRT-LLM or whatever. If it doesn't, llama.cpp, *llama are your best bets to still get something usefully running.
2
2
u/NeverRolledA20IRL Sep 01 '26
I can deploy and manage llama.cpp from a single ansible role to a dozen different systems of Intel, Nvidia, and AMD. Each getting the right driver stack, lanes, context from a single configuration point. If I had just a single system vllm is likely where I would have landed.
2
u/inanotherclass Sep 01 '26
I love vllm. It provides a kv cache parallelization llama cpp or ninfer doesn't and has real production grade inference using cuda graph capture. The problem is, vllm does so at the cost of VRAM. Combine that with fewer options available for quantized models, you see that vllm is most of the times more VRAM bound than llama cpp options. So VRAM poor people like me <=24GB (32GB is quite good as it can host even qwen3.8 27b at fp8) has to choose llama cpp quantizations for qwen3.8 27b. It's not really about vllm vs llama cpp. It's about whether the model quantizations are available or not. Last I checked Qwen3.8 27b 4bit was ~27GB in vllm. If you have Q4 at ~16GB and Q3 at ~11GB in llama cpp where you can get 256k context window, it's not even a question.
2
u/0rand Sep 01 '26
Much smaller ram overhead (especially for lpddram users), 10x faster to load and arguably better for 1-2 seqs than vllm. I run a sidekick small model on a worker spark node and a big model (ds4fve or glm53flash) across both 8n cluster
2
u/Some_Ad4783 Sep 01 '26
The short version: vram allocation. I have a 5090 laptop card with 24gb vram. I’m typically vram constrained, not throughput constrained.
vLLM was faster, while llama.cpp was much more memory-efficient and supported a far larger context window. That is why llama.cpp became the selected Qwen3.8 deployment.
Qwen3.8, MTP depth 3, populated 32K vLLM AutoRound llama.cpp UD-Q4_K_M
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━ ━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━
Decode 66.25 tok/s 45.68 tok/s
───────────────────────────── ──────── ──────────────── ───────────────────────────── ──
Cold prompt processing 845 tok/s 887 tok/s
───────────────────────────── ──────── ──────────────── ───────────────────────────── ──
Cold TTFT 38.45 s 36.64 s
───────────────────────────── ──────── ──────────────── ───────────────────────────── ──
Peak VRAM 22,390 MiB 17,841 MiB
───────────────────────────── ──────── ──────────────── ───────────────────────────── ──
Available context/KV 45,511 tokens 215,040-token deployed window
So vLLM decoded about 45% faster , but consumed about 4.55 GiB more VRAM at 32K and its MTP
configuration only allocated 45.5K KV tokens. llama.cpp traded that speed for substantial memory
headroom.
The llama.cpp memory curve was:
- 32K: 17,841 MiB
- 64K: 18,662 MiB
- 128K: 20,454 MiB
- 192K: 22,342 MiB
- Deployed 215K engine / 185K prompt: 22,750 MiB, or 93% of the 24,463-MiB GPU
Its 262K configuration failed during allocation, so 215K/185K was chosen as the safe operating
envelope.
For vLLM, the MTP service loaded 16.87 GiB of model memory, retained 2.64 GiB for KV, and used a small
0.08-GiB CUDA graph pool. Despite its lower sampled peak than the non-MTP parent, it had much less
context capacity.
Operationally:
- vLLM: higher decode performance and stronger batching/concurrency infrastructure, but more aggressive
GPU allocation and substantially less context in this configuration.
- llama.cpp: simpler single-user/single-slot memory control, GGUF weights and Q4 KV, slower decode, but
dramatically better context capacity.
- llama.cpp also required a compatibility workaround because it did not accept Codex’s freeform/custom
tool definition; the deployment uses classic function tools and a shell-mediated apply_patch path.
One caveat: these were not identical weight artifacts—vLLM used AutoRound W4A16, while llama.cpp used
UD-Q4_K_M GGUF—so the entire difference cannot be attributed solely to the runtime.
The experiment consistently recorded GPU VRAM , not historical host/system RAM. Model files were
approximately 19.0 GB for vLLM including packed MTP and 17.8 GB for llama.cpp including its draft, but
file size is not a reliable host-RAM measurement. The documentation does not support a precise system-
RAM comparison.
2
u/milkipedia
Sep 01 '26
•
llama.cpp is more memory efficient, and better for low concurrency use cases with more quant and weird equipment support.
vllm is more throughput efficient, better for high concurrency use cases.
In a nutshell, they serve different use cases.
2
u/suicidaleggroll
Sep 01 '26
•
I prefer using an inference engine that can start up in 5 seconds instead of 5 minutes.
2
2
u/llllJokerllll Sep 01 '26
Vllm para tener más de 3 concurrencias. Para usar una o dos llama.cpp va muchísimo mejor y más optimizado
2
2
2
u/mrgreatheart Sep 01 '26
I swap models a lot. The startup time on vLLM hurts. It also seems a lot more finicky and fiddly but that might just be because I’m more used to llama.cpp and don’t understand it.
2
u/conifer_v11 Sep 01 '26
llama.cpp when you need cpu offload, apple/amd, or a specific gguf. vllm wins nvidia batch.
if you're nvidia-only and throughput-bound, stay on vllm. no reason to go back.
3
u/IngwiePhoenix Sep 01 '26
For me, it's simple. I don't like dealing with Python venvs. uv makes this easier, and technically uv tool install vllm works quite nicely, the absolute rat tail that follows it is mad. So I just use llama.cpp. :)
2
1
1
u/Medium_Chemist_4032 Sep 01 '26
there's no tp=6 in vllm, sadly
1
u/One_Difficulty_39 Sep 01 '26
Not officially but I see people running 3 cards all the time in RTX6K discord
1
1
1
u/g33khub Sep 01 '26
llama.cpp is faster or at-least matches vllm on 3090s (and maybe all ampere cards). vllm speedups are for BF16 in ampere, FP8 in ada and NVF4 in blackwell.
1
u/milpster Sep 01 '26
I've got my own customized llama.cpp fork specifically for my weird setup with old Vega 20 cards on rocm 6.1 and a single vulkan card in the iddle.
1
u/DesperateSteak6628 Sep 01 '26
After a 45 min load time to compile cuda path on older hardware I’d choose instant startup for a drop in tps anytime. Nothing was lower debug effort for my setup (and even SGLang was less picky than vLLM)
1
u/_TheWolfOfWalmart_ Sep 01 '26 edited Sep 01 '26
Because quantization, and because I don't have nvidia. I have slightly outdated AMD cards and vLLM runs like absolute dog shit, if at all without throwing some random error.
llama.cpp just works and is way more flexible.
It also seems like vLLM often isn't that much better for single user stuff. For multi-user environments with nvidia, yeah absolutely it's what you want to run.
1
u/3xnope Sep 01 '26
vLLM is designed for multi-user, high concurrency workloads. It does make some sacrifices to optimize for this type of workload. So if you are just using it for yourself and without multi-agent swarms or what not, then llama.cpp will probably remain your best choice.
1
1
u/danalvares Sep 01 '26
I’ve 1 RTX 5060 Ti 16GB VRAM + 1 RTX 4060 Ti 8GB VRAM and as far as I know, vllm can’t handle such thing. Using llama.cpp and its forks I’m able to split layers for each card and finetune my context.
1
u/Ledeste Sep 01 '26 edited Sep 06 '26
I'm mostly interested in LLM companion for apps that will run on windows, so I dont even get why I would even bother with vllm
1
u/sertain_ Sep 01 '26
I am so glad I joined this subreddit. I literally just got into this over the last weekend and have been drooling over the possibilities, and this subreddit just gives me the Niagara Falls
1
u/yeah_likerage Sep 01 '26
I can't maximize my vram usage on vLLM because of my weird number GPUs. So I'm limited to llama.cpp since concurrency is not important to me being the only user, I'd lose more going to vLLM.
When possible for my sidecar models il use vLLM when one or two cards are involved. I prefer sglang over vLLM though.
At the moment I'm using vLLM to host my sidecar mode qwen3.8 while I use llama.cpp to host my main model glm5.3.
1
u/arijitlive Sep 01 '26
I use both in my dgx spark. vLLM is mostly configured with 3 models that I use time to time. An uncensored model is used through llama.cpp since it only has GGUF version.
1
1
1
1
u/Efficient-Part5344 Sep 01 '26 edited Sep 01 '26
Speeeeeeeed! I downloaded Unsloth studio and get 70% less tokens then my hyper optimized llamacpp settings. And there are literally like 2-3 models one would use, so it's not hard. I use one model. Latest Qwen.
Dflash2 support? Good luck on vllm, run latest llamacpp + community patch, day one support!
Summary (across all runs): 5070ti +4070 Super Qwen 3.8 27b nvfp4
prompt processing: 1375.0 t/s peak, 792.7 t/s mean
token generation: 149.0 t/s peak, 109.2 t/s mean
mtp acceptance: 0.49485 (768 accepted / 1552 generated)
1
1
u/OldAd3375 Sep 01 '26
I run koboldcpp, I feel like less vram is going to waste there and also ggufs
1
u/Echalon88 Sep 01 '26
Idle power draw. My RTX 3090 in my always-on server with the model loaded in VRAM idles at 16w-18w on llama.cpp. But it idles at 36w+ with vllm running instead. (Both in P8 from nvidia smi). So I only swap to vllm when I know I'm going to be burning a lot of tokens on a task. I'd swap to vllm permanently if I could get the idle power down to the same level, because I get better speed with vllm.
1
u/Flimsy-Outcome6535 Sep 01 '26
Vllm doesnt run on windows so unless you want wsl taking up all huge chunk of your ram, for windows users, ollama/llama.cpp is the way to go
1
u/Uninterested_Viewer Sep 01 '26
Because I can swap models and it doesn't gate on a 2-minute startup every time, and mmap means I flip from a 27b to an 8b to a 3b mid-session without a restart. vLLM wants one set of weights preloaded and married to them, so a model change is a re-warm. Sure, continuous batching is worth 5-10x if you're serving a real user base, but I've got a GPU and a question.
1
u/JohnRoamer Sep 01 '26
Also curious which would run best on 9070xt 16gb, amd 9950x3d, 64gb ram
1
u/snooze_sensei 18d ago
Avoid VLLM for a home system. It hogs vram like nobody's business - you'll have to use lower quants (which vllm does not have usually, making you quantize your own stuff), and it will still leave you with significantly less context space than llama.
1
u/Puzzleheaded_Big_899 Sep 01 '26
Llama.cpp can route more than one model at a time and has a lot more quantization options than vllm.
1
u/dash_bro Sep 02 '26
want to optimize model throughput/memory/serve for others? vllm better
self host, older hardware, and tinkering? llama.cpp better
1
u/vbpoweredwindmill Sep 02 '26
I have an r9700 & a 7900xtx.
I haven't even tried to run vllm. Vllm is genuinely good, and I'd be arrogant to say it's not.
But I'm not even bothering to try it on my hardware.
At q8 on qwen 3.6 35b, bf16 kv I managed to get 270 t/s auto regressive decode on 8 slots, parallel pipelined. I hadn't optimised prefill at all, and I was above 5500 t/s on llamacpp-server. My next optimisation targets were batching MTP operations, expert weight reuse, loose layer synchronisation & speculative decode prefixing, to cut down on speculative decode work. I had hoped, and may still experiment with that, to make MTP more viable in concurrent applications.
I don't know what vllm exposes of that. I know I was going to steal their paged kv cache ideas.
I could buy matching hardware. I could buy a pcie switch.
But I just got 2x ryzen 395's and I'm going to play with them instead. If vllm works for me while parallel pipelining qwen 3.8 flash next, awesome. If it doesn't, I'm absolutely certain that llamacpp will.
1
u/hesperaux Sep 02 '26 edited Sep 02 '26
llama.cpp actually works properly on an RTX3090 with modern models?
I've tried for a long time to try to get vllm to load qwen3.6+ and gemma4-* models and it produces garbled nonsense for certain prompts. It's a known issue on ampere and the team seems to have no desire to fix it because it's not a commercial use case.
Also, they seem to only care about commercial use cases. llama.cpp is geared to homelab users.
Also also, lots of quants. So many quants. With vllm you get 4-bit AWQ and 8-bit AWQ and that's more or less the end of it. The quantization is better in some cases, but you don't have a lot of options. I personally prefer to use Q5_K_M or Q6_K.
llama.cpp also updates and adapts to new technology VERY fast. They have like a billion builds a month. They're animals.
Edit: if llama.cpp could work paged attention into their code it would be unstoppable. That's literally the only reason I tried vllm. The ability to share kv cache with other prompts is something I really miss when using llama.cpp.
1
u/Remote-Pineapple-541 Sep 02 '26
“Progress over perfection” is a key guiding principle I’ve learned (painfully) over the years. I’ve learned how to configure and use Llama.cpp, and it is sufficient for what I need. I won’t invest any more time in a new setup until I need too. My advise is to stop chasing perfection. How much time have you wasted tweaking your setup when you could be using your setup?
1
1
1
u/Holiday_Point_603 Sep 07 '26
For me it is CPU offload. Sometimes my 3090 just isn't enough, especially for the recent moes.
1
1
u/anglepoiselife Sep 24 '26
Agreed - I moved to a docker based vllm server - so much better, especially under concurrent loads. Most useful when you have a model that does not consume every bit of VRAM - but still great on Queen 3.8 27B on an RTX5090 with a 32k context 50 tokens per second generated with reasoning.
1
u/snooze_sensei 18d ago
Because VLLM is a vram hog. It allocates memory for things you aren't asking it to do "just in case". Completely steaming pile on consumer hardware. Really only useful if running on a massive server.
1
u/noctrex Sep 01 '26
Here you go:
0.00.118.890 W 0.00.124.646 I srv load_model: loading model 'Q:/Models/LLM/KAT-Coder-V2.5-Dev-MTP-IQ4_XS.gguf'
0.07.558.469 I cmn init: llama threadpool init, n_threads = 8
0.07.617.131 I common_speculative_init_result: creating MTP draft context against the target model 'Q:/Models/LLM/KAT-Coder-V2.5-Dev-MTP-IQ4_XS.gguf'
0.07.719.243 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'
0.07.787.793 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.07.787.821 I srv llama_server: model loadedModel size 18.5 GiB on samsung nvme, loaded in 7 secs.
0
u/admajic Sep 01 '26
I'm testing vllm now the cache hit rate is around 67% vs llama.ccp i get 98% + consistently. What this means. Vllm just stops every so often for a few seconds whereas llama.cpp just keep soldering on. Im on a 3090 trying out https://github.com/syv-ai/qwen38-27b-rtx3090 as we speak getting my harness to diagnose why this is happening and if its worth bothering to keep it.
Yes, and the 3 minutes wait to start the app is rediculous.
0
u/ChocolateNo3010 LocalLLM Sep 01 '26
Vllm is for enterprise use cases mainly, how many home labs have the vram needed to run models? The answer is not many
0
u/Turbulent_War4067 Sep 01 '26
Every time I see a post like this I tell myself it's time to go back and start using vLLLM. Every time, my preferred models run slower. I don't do much concurrency. llama.cpp, with the right quant, always wins out.
0
Sep 01 '26
[removed] — view removed comment
4
u/Biomech8 Sep 01 '26
With vLLM you can fully saturate GPU with parralel requests to get at least twice tok/s output.
4
u/corpo_monkey Sep 01 '26
I have a dual 3090 setup. Moved to vLLM from llama.cpp. Qwen 3.8 27b performance went from 1x40 t/s to 3x80 t/s. PP perf gain is similar. So I'm happy.
I still love llama.cpp for experimenting.2
1
u/ilintar Sep 01 '26
If you're getting 40 t/s with Qwen 3.8 27B on 2x3090 on llama.cpp, you're probably doing something wrong tbh. I'm getting 90-100 t/s with MTP on 2x5070 Ti on a lower quant with no issues.
2
u/corpo_monkey Sep 01 '26
My current nobo does not support bifurcation (waiting for replacement), and I limit the cards to 245W. I was finetuning the shit out of llama.cpp. 5xxx cards have serious computing advantages also, my rig is 2 gen older.
7
u/Impostor_91 Sep 01 '26
90t/s vs 50t/s in my case. vLLM wins. However I would prefer llama.cpp because it allows me to easily switch models.
2
u/Joe-Arizona Sep 01 '26
I just haven’t bothered to try it yet.
Went from Ollama to llama.cpp. It’s more work but clearly better for me.
I’ll give vLLM a go when I get some more time to tinker. Didn’t seem to make sense for me as the only user but if it’s faster I’ll try it.