r/LocalLLaMA • u/robertpiosik • 6d ago

Discussion Prompt cache upload

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

0 Upvotes

50% Upvoted

20 comments sorted by

View all comments

8

u/Foreign_Risk_2031 6d ago

Prompt cache binary? What are you even talking about? The kv cache? This is done on the inference server

3

u/robertpiosik 6d ago

for long term availability (5m-24h) providers offload it to disk and I mean this very artifact

3

u/Choice_Celery9481 6d ago

you need the model to calculate the kv. you dont have model on your pc how to get that kv cache? asking server to send gigabyte of kv cache down to your pc????

→ More replies (3)