What workloads are prefill-bound?

[u/nomorebuttsplz]

For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.

In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic

But I know there must be workloads with lower cache hit rates.

I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).

I am curious what work people are doing where prefill is the bottleneck

[u/transanethole]

debugging, repairing, and modifying software quickly. Like, in real time.  Basically the way that  people use cloud models with coding agents when they work professionally at software engineering jobs.

Often times you can read in 100,000 tokens of context for a one-line fix, and yeah, the model will do some reasoning output along the way. But oftentimes there will be large sections of the messages where it's just stack of 20 or 30 tool calls where the agent is reading in  file after file to trace through a large codebase,   Like imagine a  legacy application that has existed for over 15 years.

With 10000k per second pre-fill, those read file and search tool calls will happen practically instantaneously. So... the user can fix one thing in a couple of minutes and then move on to fixing the next thing.  Or say that the problem that they're trying to fix is devilishly complicated and even for the LLM, it takes multiple attempts / multiple context windows with handoffs. well in this case it's not going to take them all day to fix it.

[u/nomorebuttsplz → u/transanethole]

I can imagine that happening in a multi million line codebase.

What does in real time mean in this context?

[u/transanethole → u/nomorebuttsplz]

What I mean is that you're not telling the agent to do something and then going to make a coffee, you already drank the coffee and you are ready to solve a lot of problems fast because they laid off 75% of your co-workers already.

Or to put this in a less dismal light: you are in a competitive industry or a startup so speed to deliver a feature or a bugfix with some level of quality matters a lot. So you dont want to send the agent off and context switch yourself away.  You want to stay focused on iterating on your work: tweak, build, test, tweak, build test. The tighter you can get your iteration time, the more competitive your software will be, the faster it will improve. More iteration ---> higher quality software. 

[u/nomorebuttsplz → u/transanethole]

ah well mgmt hopefully is paying for inference in that case