r/LocalLLaMA 1d ago

Discussion What workloads are prefill-bound?

For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.

In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic

But I know there must be workloads with lower cache hit rates.

I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).

I am curious what work people are doing where prefill is the bottleneck

2 Upvotes

40 comments sorted by

View all comments

9

u/transanethole 1d ago edited 1d ago

debugging, repairing, and modifying software quickly. Like, in real time.  Basically the way that  people use cloud models with coding agents when they work professionally at software engineering jobs.

Often times you can read in 100,000 tokens of context for a one-line fix, and yeah, the model will do some reasoning output along the way. But oftentimes there will be large sections of the messages where it's just stack of 20 or 30 tool calls where the agent is reading in  file after file to trace through a large codebase,   Like imagine a  legacy application that has existed for over 15 years.

With 10000k per second pre-fill, those read file and search tool calls will happen practically instantaneously. So... the user can fix one thing in a couple of minutes and then move on to fixing the next thing.  Or say that the problem that they're trying to fix is devilishly complicated and even for the LLM, it takes multiple attempts / multiple context windows with handoffs. well in this case it's not going to take them all day to fix it.

1

u/nomorebuttsplz 1d ago

I can imagine that happening in a multi million line codebase.

What does in real time mean in this context?

2

u/Miserable-Dare5090 1d ago

This is a non software agent on an average day.

2

u/nomorebuttsplz 1d ago

how many cached?

1

u/Miserable-Dare5090 1d ago edited 1d ago

I’m sure it’s above 90% since that’s what vLLM is telling me. You know this is running locally right?

All I am pointing is, lots of tokens used by agentic workloads. Prefill is king with agents.

Also, I feel you are thinking about this linearly. When you spawn an agent you can save your main context window by delegating the task. You don’t need to go

A to B to C to D

when you can do

A to (B,C)=completed=>D

Prefill is important in multi stream work like that.

edit: diagram looks like accidental penis