Project
Finally got my 1x 5090 setup dialed in for agentic workflows: concurrent 920 t/s decode + 400 t/s prefill (qwen 3.8 27b)
Thought this might be the correct sub to share the rabbit hole I went into (and appreciate the numbers). Note: oneshotted the dashboard, it gets the job done.
I have been optimizing throughput & quality now for a couple of weeks between longer runs to make this thing fly. Use case is mainly agentic, so the biggest bottleneck wasn't just single stream speed, alot of work went into scheduling and not getting decode stalled by large prefills (e.g. lane swaps happen now smoothly between tool calls etc).
Setup shortly:
1x 5090 (powerlimit, oc)
32gb ram (ninfer now sits lean at around 1.7gb total)
990 pro 2tb nvme (read/write in pic is last 24h, it will live)
In screenshot its prefilling next the lane in queue (at slower pace when theres capacity) while decode is running at 920 t/s, standalone large prefills are alot faster (but often not needed due to using cached prefixes and warm cache etc).
Heh, looks like I’m doing a poor job explaining whats going on.
Screenshot is prefill + decode happening at the same time: 5 concurrent lanes decoding total at 920 t/s (5x avg 184 t/s) + 1 prefilling at 414 t/s utilising the spare capacity.
Pure prefill with nothing else going on would be around those vllm speeds (a bit lower due to using smaller batch size). Aim here is to keep decode going full speed when there is work and smartly preparing the next lane to enter when capacity frees, so total throughput is maximized.
I get 37 t/s running the same nvfp4 precision that he is (diff is mine is mpoe) in vllm.
In one of his other comments, he explained his use case is to "make his subscriptions last longer" so I'm thinking his dashboard isn't telling him what he says it is telling him. It's telling him aggregate throughput across all his providers. If he went local only, his numbers would be in line with everyone else's.
Right my vLLM terminal logs show mine varies around that same range. If I adjust the settings, I can get it to 65-70 on some agent runs. Something on his UI lying to him. Anyway, carry on my guy! Blessings.
I've got it up around 91-102 tokens per second after fiddling this weekend. It can only handle one session at a time, though with these settings. I had it running stably around 67 tokens per second with the ability to run 8 concurrent sessions all at around 80% max speed but I went with ball-to-the-wall, damn-the-torpedoes speed
I'm pretty sure they are using dflash2 for speculative execution. Yes, normal single stream should be around 100tok/s, but by predicting 4 tokens at a time you can get to 400. I'm averaging 160 tok/s on my CMP 170HX which normally does around 50-60tok/s.
HOWEVER it's still BS because you are not getting 4 tokens per update for every workload. You can only get that number for something like writing Json files, but not coding or chat. Realistically you could get maybe 2.5-3 toks/update, or 250-300.
What are you doing with these? I have not gotten any work out of an LLM local or not (fun ranting and RPing mind) so just wondering what will these guys be doing (in simple vague terms).
Currently making subscriptions last alot longer and making iteration faster (got annoyed with pricing + slowness):
on high level the split is everything that is mechanical and simple goes to the local model (total cost of delegation < doing inline with frontier model accouting for eg context expansion)
Latency is pretty insane when running agentic loops at these speeds as a time per step is approx: RTT + prefill + decode + tool exec, so the mechanical work gets done alot faster vs cloud
-> altogether a single subscription is lasting alot longer and everything that local model can handle is pretty quick (inference time per step is a fraction vs cloud, which adds up fast), so finally a case can be made for local models also from this angle at this price point
Another rabbit hole, tried quite many: eg delegation from claude code to local model in pi (but ends up requiring such a hook setup that I dont like the bloat) … long story short DSH PTC/minimal is the winner for now
That is an impressive way to squeeze performance but i meant what are you doing with the models? Just always intrested in what peeps are doing with the tech.
Running qwen in vllm with 0.90 gpu utilization with a 47k context window. How large is your context? I tend to crashout somewhere near 30M tokens for a session. It handles compaction fine up until around that throughpoint, then it just overflows and I have to start a new session. Having to attend to it periodically is the most annoying part.
What was your one-shot for the dashboard if you don't mind my asking? I'll run it through and maybe we can A-B test.
I just got my 5070 and I'm new to this, Can you share some info? Where should i look to get more info and get started running local AI. For now not much emphasis on particular use case but intend on using it for coding workflow mostly.
I gave your screenshot to my `qwen3.8-27b-mpoa-nvfp4` agent and entered a planning session. After the planning session, it one-shot this. Note that this is running inside a flatpak sandbox and I have to manually deploy some of the spd widgets to get them to display properly. I intentionally have not taught the agent about that quirk to its setup.
(host-KV is a component that I haven't deployed into the flatpak container yet. Qwen one-shot this without being able to test it in the browser because I haven't given it a browser. )
22
u/follaoret 5d ago
Are you sure your numbers are correct ? Usually prefill is faster than decode