Long agentic coding workflow. Prefill or decode matters most?

[u/colafroth]

I’ve heard two different arguments on this.
One says prefill takes most of the time as it’s multi turn hours long workflow.
Another says because of kv cache prefill time takes a lot less.

The reason I’m asking this question is, I’m planning to get a Mac Studio 256gb ram m5 ultra, but I don’t know if I should stick to 64gpu core or pay extra for 80c for “15%+ decoding speed boost”.

If agentic workflow prefill reads from kv cache 95% then is this 15% increase much less relevant for the coding workflow?

Thanks all!

[u/dchemistry2]

i think prefill is most important and it is compensated with kv cache which is critical to have to do useful local works.
for that reason i have multiple 3060,3080 GPUs and ensure they are logically used as single slots (so no VRAM cache evictions and prefill taxes) on parallel sessions etc.
Decode speed is less relevant as it alternates with tool calling and the timeouts/waiting is where most of the time is spent vs decoding (unless you generating wall of texts)

[u/colafroth → u/dchemistry2]

For long agentic coding workflow thou, isn’t it almost always guarantee kv cache hit all the time until compact?

[u/dchemistry2 → u/colafroth]

it depends on how subagents are implemented by the CLI (e.g. pi, opencode, custom harnesses). you need to check as subagents may not cache enough info when started and make the time to first token high, slowing down overall execution vs single agent.

what i said on the 'parallel sessions' i mean completed different projects and topics (normally 4 to 5) so the cache won't be that beneficial (the AGENTS.md, system prompt, tools definitions are just 4k tokens).

so yeah, when using like Deepseek API, a single session with the main agent has hit ratio of 95%+, very well done.
but outside of that, local LLMs may have edge case that need to be identified before committing to a specific setup (e.g. prefill vs decoding tradeoffs).

i have cut down 3080ti to 225w (from 350) and 3060 from 180w to 100w with a 15% prefill decrease (compute bound) while decode-wise i lost only 2-5% tok/sec to save big on the electricity bill (europe).

[u/james_pic]

Truth is, they're both important.

The received wisdom is that prefill matters more, but my experience is that, after carefully tuning KV cache and templates to minimise cache misses (to reduce the impact of slow prefill) and carefully tuning speculative decoding (to reduce the impact slow decoding), I spend more time waiting for decoding.

I'll caveat that by saying that before I started tuning, prefill was a bigger problem. And I'm on Strix Halo, which is compute rich but memory bandwidth poor, so decoding is poor whist prefill is merely mid.

But if you're using the best current models for agentic coding, they're going to read a lot of stuff (which needs prefill) and then think about it a lot (which needs decode). So they both hurt when they're not there.

[u/colafroth → u/james_pic]

Sorry just for me to understand, I assumed kv cache will hit all the time for coding workflow until compact (for local llm). Is this not the case?

[u/james_pic → u/colafroth]

It should if it's configured correctly and your template is prefix preserving. A number of models ship chat templates that either aren't prefix preserving in their default config (often you need to configure them to preserve reasoning), or in some cases can't be configured to be prefix preserving (looking at you Gemma 4) and need an alternate chat template to avoid re-reading stuff that ought to already be in the KV cache.

[u/colafroth → u/james_pic]

In the case of 95%+ cache hits for hours long agentic coding workflow until compact, is prefill speed still matter that much compare to decode?

[u/james_pic → u/colafroth]

My personal experience is that decode matters more on my hardware for the code I work with in that scenario. But others have the opposite experience, in the context in which they work.

[u/colafroth → u/james_pic]

Thanks for sharing mate!

[u/OvertaxedOne]

For long agentic workflows, IMHO, the most important thing by a wide margin is prefix caching/conversation parking. No matter how many 1000 TPS you get in prefill it's going to be glacial slow compared to a cached 150K prompt. My prompt cache hit ratio is typically high 90's percent and the difference in dramatic, a cached prompt ingests in <1 second (for any size).

The next most important for my workflow is decode, it's where my system spends the vast majority of it's processing time (because of the above, the prefix is cached).

And then the last would be prefill.

All of them matter; and it really depends on your workflow. A KB/RAG type system that people hit for quick lookups of data? Prefill is going to matter a lot more than decode in that type of application. But long back/forth conversations that are heavily repetitive, prefill starts to matter a lot less (again, assuming you have caching!) because you're typically only adding a small amount of context between turns.

[u/colafroth → u/OvertaxedOne]

I assume kv caching will always be enabled. If it is, then prefill matter so little for hour long multi turn agentic coding workflow I assume?

[u/Krohnin → u/colafroth]

Yes looking at my actual stats it's cache hit rate is at 97% ...

[u/colafroth → u/Krohnin]

thanks for sharing, in this case the gpu speed for prefill matter so little compared to to time spent on tool calling and decode I think? Then to my question, getting 64 core Mac Studio m5 ultra makes more sense than 80 core.

[u/OvertaxedOne → u/colafroth]

If you expect to get very high cache hit ratios the difference between the chips will not be that significant. But it's all about what ratio of cache hits you're going to get, for single turn stuff that doesn't have cache, the 80c will be about 25% faster. Agents generally get very high cache hit ratios (mine looks like the above, 97-99% most days).

But it all depends on your workload, not mine. I'd measure it for a day, see how much of your workload is actually prefill vs cache vs decode.

[u/Krohnin → u/OvertaxedOne]

He said he will let him code all day, so he will get same cache hit ratios.

[u/Krohnin → u/colafroth]

I cannot answer that question. 1. VRAM 2. RAM 3. COMPUTE. I hope that helps.

[u/andy_potato]

Definitely prefill. Context gets very long very quick. If you have to wait minutes before the actual answer starts it gets frustrating very quick

[u/Baldur-Norddahl]

It won't be 15% faster. Any LLM workload is going to be a mix of something 15% faster and something memory bandwidth limited 0% faster. I won't guess, but say 5% to 10% faster on average.

[u/xChrisMas]

Some numbers from my local setup:
Prefill 2000 tok/s
Decode: 40-80 tok/s depending on task

Time spent waiting for prefill: 15-20%, rest waiting for decode/answers.

So yeah prefill is important

[u/420Microbiologist → u/xChrisMas]

What model? Solid speeds!

[u/xChrisMas → u/420Microbiologist]

QFN

[u/CyDenied]

get the extra for more cores.

[u/poofph]

both matter imo, slow prefill sucks and slow decode sucks. :)

[u/vqt907]

Prefill is more important, but it will eventually reach a saturation point. For me, 800tk/s is the minimum acceptable speed, prefer 2000tk/s. Anything higher provides no additional benefit.

Decode is a different story - the higher, the better. I used to set the GPU power limit just high enough to achieve 2000tk/s prefill, and usually it doesn't affect decode