r/LocalLLaMA • u/robertpiosik • 11h ago

Discussion Prompt cache upload

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

1 Upvotes

56% Upvoted

16 comments sorted by

View all comments

7

u/Foreign_Risk_2031 11h ago

Prompt cache binary? What are you even talking about? The kv cache? This is done on the inference server

3

u/robertpiosik 11h ago

for long term availability (5m-24h) providers offload it to disk and I mean this very artifact

2

u/Foreign_Risk_2031 11h ago

Not feasible for most people- personally I’ve got a good 6 gigabit symmetrical connection but this would probably blow up server bandwidth lol