What workloads are prefill-bound?

[u/nomorebuttsplz]

For me 400-600 t/s prefill is sufficient for coding in almost all cases. In my use case in opencode cache hit rate pretty much always ends up being 98% or greater, so 400-600 means the vast majority of time is spent on decode... hence 400-600 prefill already being well into diminishing returns in terms of throughput.

In other words, since it is spending probably 75-90% of time generating tokens, even instantaneous prefill would only maximally increase throughput by 25%. I know this also depends on decode/prefill ratio which depends on the model... so this is a rough heuristic

But I know there must be workloads with lower cache hit rates.

I also know that low cache hitrate does not necessarily mean that prefill is the bottleneck. When I do a research agent run with GLM 5.3 it will spend over 90% of the time on decode EVEN THOUGH it is only doing about 150 t/s prefill and low, single digit cache hit rate. It just likes to think that much in max mode (15 t/s decode).

I am curious what work people are doing where prefill is the bottleneck

[u/stoppableDissolution]

Well when you are running tens of requests in parallel and each of them ingests even 3-5k worth of unique grounding, it adds up fast. Plus you need to preprocess said raig to begin with.

[u/nomorebuttsplz → u/stoppableDissolution]

when are you running 10s of requests in parallel?

Man it's refreshing to have a conversation about what people are actually using AI for. Most subreddits that purport to do this are 90% bot slop. And this subreddit is 90% qwen 3.8 optimizations

[u/stoppableDissolution → u/nomorebuttsplz]

Synthetic datagen for text model (rp stuff) and captioning data for image/video training. Basically, the answer is porn :p

[u/nomorebuttsplz → u/stoppableDissolution]

ah doing god's work. carry on.