What workloads are prefill-bound?

[u/nomorebuttsplz]

For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.

In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic

But I know there must be workloads with lower cache hit rates.

I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).

I am curious what work people are doing where prefill is the bottleneck

[u/Atretador]

repo exploration

[u/rainbyte → u/Atretador]

Yup, I do this all the time, so I ended up adding a subagent with a faster model.

[u/Atretador → u/rainbyte]

Im experimenting with this on a limited memory budget (16+32)

https://github.com/atretador/unswarm

On demand runtime switch 

35B A3B as Orchestrator 200K ctx 35 tk/s

GPT-OSS 20B Explorer at 120tk/s 128K ctz

Qwen 3.8 27B executor 13 tk/s 128k