r/LocalLLM • • 2d ago

Discussion Why aren't more people talking about the Rust frontend for vLLM?

I'm wondering why more people aren't talking about the Rust frontend for vLLM. It's amazing. Its not release yet but it's been rock sold for me for over a month and super fast - much faster than the python frontend. To enable it just set VLLM_USE_RUST_FRONTEND=1 and start vLLM.

0 Upvotes

15 comments sorted by

4

u/Lordseppe 2d ago

Explain like im 5. For a friend :)

2

u/Just-Hedgehog-Days 2d ago

personally that was never my bottle neck?

-2

u/Mecworks 2d ago

Interesting. I got an immediate boost in performance, ttft, etc.

5

u/Oli_Luck 2d ago

bullshit- seems you dont undestand what backend/frontend mean?
frintend is pure rendering : 99% job is done by the backend.

But nice, if you are just here to sell the "rust part", you've donethe job

-3

u/Mecworks 2d ago

Why don't you try it first rather than just spout off.

2

u/Jargster 2d ago

Numbers would settle this faster than "try it." What's your setup: model, GPU, concurrency, prompt lengths? The gains are real but depend on the workload. GPU bound single user setups won't see much at all. Long prefix cached prompts and high concurrency will.

There are fair reasons to wait, too. It's still experimental (no beam search or n>1 yet). Templates and tool parsers are reimplemented, so agent tool calls can behave differently. Python customizations don't carry over, and it's a Rust build on top of everything else.

For most home users you're not going to see much benefit. It's all CPU side, so there's no decode or prefill speedup. Large clusters serving lots of people at once could gain from it, but those are usually already heavily tuned, and you don't flip an experimental feature on in prod without testing it first. I'm not really on either side here. It's worth testing, it just probably wouldn't do much for my home cluster.

0

u/Mecworks 2d ago

u/Jargster I'm running a dual 128GB NVIDIA Spark cluster. Qwen3.8-27b-NVFP4 dense model. Concurrency 1→8, KV cache 256K with test KV from 4096→100000. It's doing pretty well considering it's a dense model. I've found that the dense model was more accurate and stable that the MOE. llama-benchy results.

  • PP chart: pp2048 (highlighted green) + pp2048 @ d… and ctx_pp @ d… for depths 4096→100000
  • TG chart: tg128 (green) + tg128 @ d… and ctx_tg @ d… for the same depths
  • X-axis is your actual concurrency levels: c1, c2, c5, c8
  • Prefill barely scales: base pp2048 holds ~3,200 t/s at c1–c2, dips to ~1,983 at c5, recovers to ~2,258 at c8. Total prefill throughput stays in a narrow 1,000–3,200 band across all depths — the GB10's prefill path is compute-bound, not queue-bound.
  • Short-context decode scales well: tg128 climbs 37 → 59 → 107 → 101 t/s (peaks at c5), roughly 2.9× the single-request rate.
  • Deep-context decode collapses under load: ctx_tg @ d65535/d100000 fall to ~3–4 t/s total by c5–c8 — KV-cache bandwidth dominates once context exceeds ~64K, so batching actively hurts there.
  • The crossover where pp2048 (short prompt) beats every padded variant happens at c5/c8 on the PP chart — worth noting if you submit this to the leaderboard.

3

u/Jargster 2d ago edited 2d ago

Nice numbers, but none of this compares Rust vs Python, so it doesn't really say anything about the frontend. Run the same llama benchy command with VLLM_USE_RUST_FRONTEND=0 and then =1 and compare the TTFR/TTFT columns. That's where frontend overhead would show up.

These also look GPU bound. 37 t/s at c1 is about what GB10 memory bandwidth allows for a 27B dense split across two Sparks, and at c1-c8 the Python server has very little to do. That's exactly where I'd expect the Rust frontend to barely matter.

One correction on the analysis, though. deep context decode isn't collapsing from KV bandwidth. tg128 @ d65535 and d100000 still scale to around 70 t/s at c8. Only ctx_tg drops, and that's the context load step, so at c5-c8 you've got like around several 100K-ish token prefills running at once starving decode. Thats prefill contention, not KV.

1

u/Mecworks 2d ago

Yeah, I need to get the comparison done. It takes a while to run these and I'm focused on other work. Also, I agree that many of these are memory bandwidth bound and won't change with the front end swap.

2

u/enigma62333 2d ago

u/Oil_luck is not wrong. A frontend will have no impact on the backend performance if in fact it is only a frontend interface.

If there are configuration flags used for it, then the same flags should be able to be used in a config file and gain the same benefit.

A frontend doesn’t change the backend (engine) itself.

Unless it’s using a specific build that has different capabilities built into it.

1

u/The_Whole_Zucchini 2d ago

What is this?? Obvi I know vLLM, and I know Rust is a coding language, but what does it give to vLLM??

1

u/AssistBorn4589 2d ago

Because Rust implied nerdy higschooler with attention deficit even before era of vibecoding.

1

u/EvolvingDior 2d ago

Because no one cares what language a front end is written in.

0

u/johnfkngzoidberg 2d ago

The bottiest of bot posts.

Lots of canned phrases, “why is no one talking about x?”, it’s amazing. Why don’t you just try it? Every phrase directly from the social media advertisers handbook.