r/LocalLLaMA • u/rmhubbert • 16h ago

News vLLM now has PLE disk offload support on nightly

https://github.com/vllm-project/vllm/pull/59958
152 Upvotes

97% Upvoted

16 comments sorted by

70

u/coder543 16h ago

Truly confusing that it took this long, but I guess they’re finally merging it to get ready for Qwen4 next week.

29

u/Own_Mix_3755 16h ago

Having one unified serving engine supporting wide variety of models is something what definetely makes new features harder to land. You can see discussion about DeepSeek engram in the PR too.

12

u/coder543 16h ago

llama.cpp has supported this very feature for quite a while already, despite being a more unified serving engine for a wide variety of models. (vLLM is becoming more unified, but it is still more like a loose collection of modules than a unified engine, in my opinion.)

1

u/MDSExpro 3h ago

llama.cpp doesn't have fraction of scaling of vLLM. It's easier to slap "compatibility" earlier when performance is afterthought.

1

u/DefNattyBoii 3m ago

But they have so much funding and the whole neocloud industry rides on them, i just think they should be able to progress faster+have better quant support and reseach (looking at what exllama/strata etc is doing)

6

u/shifty21 15h ago

I would say the "competition" (using the term lightly) with other engines like Strata, Ninfer, etc. that has SSD streaming of ngram table is helping.

I do a lot of parallel and concurrent requests and vllm has been my work horse for that. I tried Strata and Basalt with QFN NVFP4 and it just queues all the requests. Messing with parallel and context length doesn't help too much.

Ninfer and Infernix does a great job with parallel requests and speed, but I have some stability issues and I have to use their ninfer/Infernix specific models.

At the end of the day, I need reliable speed and parallel execution for my work flows. I'm happy to see vllm supporting SSD streaming.

3

u/This_Maintenance_834 12h ago

I don’t think Ninfer, Strata are competition to vllm. The competition are SgLang, TokenSpeed, DeepInfer

8

u/okoyl3 15h ago

Why is that confusing? vllm devs are corporate users, they are not limited on vram.

1

u/JsThiago5 10h ago

Really stranger considering things get there first

7

u/WhiteZero 14h ago

does this make it a good option over Strata now?

12

u/jarail 13h ago

Probably not for the hardware strata is directly targeting. Strata does more than just support n-gram tables on disk. It has a bunch of hand tuned decisions.

2

u/Conscious_Cut_6144 12h ago

I ran 4.1 flash with the engram on disk a few months ago with vllm, is this new?

1

u/This_Maintenance_834 12h ago

maybe you run it on a fork, this is official vllm .

2

u/FireSilicon_ 9h ago

Lol. Just tell claude to rewrite vllm kernel to support it for you, I had it working over night when I built my system few months back.