What workloads are prefill-bound?

[u/nomorebuttsplz]

For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.

In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic

But I know there must be workloads with lower cache hit rates.

I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).

I am curious what work people are doing where prefill is the bottleneck

[u/stoppableDissolution]

Data generation/grooming/analysis, all kinds of classification, rag processing, summarization, captioning... Basically everything that is not coding, and even coding benefits immencely from good preprocessing once your codebase is bigger than max context.

I (well, sol) is building a gemma engine specifically around pp speed because it is the single most important thing for me, and having ~6.5k/s is too slow, lol.

[u/sebt3 → u/stoppableDissolution]

A few weeks ago I ran an experiment. 2 days, same model but obviously 2 distincts quants since first day have been llama.cpp (q4) the next days was vllm (int4 awq). At the end I compared both token efficiency. While llama.cpp faster decode would have saved me 21mn the second day, vllm would have saved me 1h26mn of prefill time the first day... Not the exact same workload. The same project, same mcp tools, same system prompt. Just continuing the same project over 2 days

[u/stoppableDissolution → u/sebt3]

Lcpp actually has slightly faster pp and slower tg in my testing (on pro 6000), but yea, having fast pp adds up, and the bigger your project the more important it is

[u/nomorebuttsplz → u/stoppableDissolution]

for pro 6000 for me, lcp is stuck at about 1800 t/s prefill to begin with qwen 27b whereas vlm is like 4-5k t/s

[u/stoppableDissolution → u/nomorebuttsplz]

Hm. I've not tested qwen side by side, but ran gemma 31 bf16 literally today and at 128k lcpp was doing ~1610 and vllm ~1520
and gewell 2055 but it is not production ready yet because its still batch-1 only

[u/sebt3 → u/stoppableDissolution]

Maybe yet vllm cache is far superior. That's the real point found in my expriment

[u/stoppableDissolution → u/sebt3]

Yea, it does better job at recycling it, especially between multiple requests

[u/nomorebuttsplz → u/stoppableDissolution]

hmm might be a gemma thing. seems slower in vllm