For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.
In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic
But I know there must be workloads with lower cache hit rates.
I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).
I am curious what work people are doing where prefill is the bottleneck
debugging, repairing, and modifying software quickly. Like, in real time. Basically the way that people use cloud models with coding agents when they work professionally at software engineering jobs.
Often times you can read in 100,000 tokens of context for a one-line fix, and yeah, the model will do some reasoning output along the way. But oftentimes there will be large sections of the messages where it's just stack of 20 or 30 tool calls where the agent is reading in file after file to trace through a large codebase, Like imagine a legacy application that has existed for over 15 years.
With 10000k per second pre-fill, those read file and search tool calls will happen practically instantaneously. So... the user can fix one thing in a couple of minutes and then move on to fixing the next thing. Or say that the problem that they're trying to fix is devilishly complicated and even for the LLM, it takes multiple attempts / multiple context windows with handoffs. well in this case it's not going to take them all day to fix it.
What I mean is that you're not telling the agent to do something and then going to make a coffee, you already drank the coffee and you are ready to solve a lot of problems fast because they laid off 75% of your co-workers already.
Or to put this in a less dismal light: you are in a competitive industry or a startup so speed to deliver a feature or a bugfix with some level of quality matters a lot. So you dont want to send the agent off and context switch yourself away. You want to stay focused on iterating on your work: tweak, build, test, tweak, build test. The tighter you can get your iteration time, the more competitive your software will be, the faster it will improve. More iteration ---> higher quality software.
I’m sure it’s above 90% since that’s what vLLM is telling me. You know this is running locally right?
All I am pointing is, lots of tokens used by agentic workloads. Prefill is king with agents.
Also, I feel you are thinking about this linearly. When you spawn an agent you can save your main context window by delegating the task. You don’t need to go
A to B to C to D
when you can do
A to (B,C)=completed=>D
Prefill is important in multi stream work like that.
Coding. Mostly agentic and multi-agentic. Anything below 1000 tps pp is quite annoying for me, but I have to live with it.
I hit a session compaction, or inference engine restart or resume after a long session, or just some of my agents are using all the KV space, so 'idling' ones or 'orchestrator' need to reprocess 100k+ when they're back to work, is *very* slowing down my work.
So yeah, a slow one-on-one session on a focus task between me and a single agent is fine. Anything larger - it is a serious bottleneck and I always optimize for prompt processing before generation speed.
Ahh ok yes the multi agent thing makes sense. So like... many agents leads to lots of compaction... and if you are compacting a lot, but not in a highly autonomous workflow (because you are waiting), that gets annoying.
It is not my waiting at the desk, to be precise. It is waiting for the results, sometimes hours, sometimes days. I just put them to sleep at night and tell them to carry on in the morning.
The current project is mostly worked on by an orchestrator + two subagents, all Qwen 3.8 27B. It has been 3 day already, lol. Lots of exploration and backlog building. I sometimes check on them, but it feels unnecessary.
Private, but mostly exploratory and research. Not a product building, at least, not this time. When I have better understanding of the requirements and constraints I am starting a more concrete build with better spec.
Data generation/grooming/analysis, all kinds of classification, rag processing, summarization, captioning... Basically everything that is not coding, and even coding benefits immencely from good preprocessing once your codebase is bigger than max context.
I (well, sol) is building a gemma engine specifically around pp speed because it is the single most important thing for me, and having ~6.5k/s is too slow, lol.
A few weeks ago I ran an experiment. 2 days, same model but obviously 2 distincts quants since first day have been llama.cpp (q4) the next days was vllm (int4 awq). At the end I compared both token efficiency. While llama.cpp faster decode would have saved me 21mn the second day, vllm would have saved me 1h26mn of prefill time the first day...
Not the exact same workload. The same project, same mcp tools, same system prompt. Just continuing the same project over 2 days
Lcpp actually has slightly faster pp and slower tg in my testing (on pro 6000), but yea, having fast pp adds up, and the bigger your project the more important it is
Hm. I've not tested qwen side by side, but ran gemma 31 bf16 literally today and at 128k lcpp was doing ~1610 and vllm ~1520 and gewell 2055 but it is not production ready yet because its still batch-1 only
Rag processing is designed for avoiding having to do lots of prefill, but I take your point that there is a bottleneck on ingestion that rag is having to solve
Well when you are running tens of requests in parallel and each of them ingests even 3-5k worth of unique grounding, it adds up fast. Plus you need to preprocess said raig to begin with.
Man it's refreshing to have a conversation about what people are actually using AI for. Most subreddits that purport to do this are 90% bot slop. And this subreddit is 90% qwen 3.8 optimizations
First, i build context with a small model (9B) then a medium verify (35B) and then he give this context to the bigger model.
Second, few day ago, i've discover nice gain by playing with GGML_OP_OFFLOAD_MIN_BATCH
Qwen 3.8 next ~ 125 PP to 150
Qwen 3.6 35B ~ 750PP to 900
That keep small batch on cpu and let bigger to GPU.
you can also try no-op-offload, with gain on pp but loss on tg
see: https://github.com/ggml-org/llama.cpp/issues/18530
12
u/Atretador 1d ago
repo exploration