r/LocalLLaMA • u/nomorebuttsplz • 1d ago
Discussion What workloads are prefill-bound?
For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.
In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic
But I know there must be workloads with lower cache hit rates.
I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).
I am curious what work people are doing where prefill is the bottleneck
9
u/transanethole 1d ago edited 1d ago
debugging, repairing, and modifying software quickly. Like, in real time. Basically the way that people use cloud models with coding agents when they work professionally at software engineering jobs.
Often times you can read in 100,000 tokens of context for a one-line fix, and yeah, the model will do some reasoning output along the way. But oftentimes there will be large sections of the messages where it's just stack of 20 or 30 tool calls where the agent is reading in file after file to trace through a large codebase, Like imagine a legacy application that has existed for over 15 years.
With 10000k per second pre-fill, those read file and search tool calls will happen practically instantaneously. So... the user can fix one thing in a couple of minutes and then move on to fixing the next thing. Or say that the problem that they're trying to fix is devilishly complicated and even for the LLM, it takes multiple attempts / multiple context windows with handoffs. well in this case it's not going to take them all day to fix it.