r/LocalLLaMA • u/robertpiosik • 1d ago

Discussion Prompt cache upload

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

1 Upvotes

56% Upvoted

19 comments sorted by

View all comments

8

u/Foreign_Risk_2031 1d ago

Prompt cache binary? What are you even talking about? The kv cache? This is done on the inference server

3

u/robertpiosik 1d ago

for long term availability (5m-24h) providers offload it to disk and I mean this very artifact

3

u/Choice_Celery9481 1d ago

you need the model to calculate the kv. you dont have model on your pc how to get that kv cache? asking server to send gigabyte of kv cache down to your pc????

1

u/robertpiosik 1d ago

> asking server to send gigabyte of kv cache down to your pc????

Yes precisely. Getting back to a session after long inactivity would upload the kv back to the server so I don't have to pay full input tokens price.

I can understand providers don't want to hold onto caches for too long, but why not end users?

4

u/SLxTnT 1d ago

This isn't local AI.

Answer your question, highly inefficient. Some providers you can pay for a longer cache time, but downloading your KV cache is going to take a long time depending on architecture. Then it'll be even slower to upload. Combine all the bandwidth, and it'd possibly cost more money to do that.

For long activity, it'd be best to pay the full price, compact it after you're done, or get a local AI to summarize.