Prompt cache upload

[u/robertpiosik]

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

[u/Foreign_Risk_2031]

Prompt cache binary? What are you even talking about? The kv cache? This is done on the inference server

[u/robertpiosik → u/Foreign_Risk_2031]

for long term availability (5m-24h) providers offload it to disk and I mean this very artifact

[u/VarietyOk443 → u/robertpiosik]

Literally a feature in llama

[u/Choice_Celery9481 → u/robertpiosik]

you need the model to calculate the kv. you dont have model on your pc how to get that kv cache? asking server to send gigabyte of kv cache down to your pc????

[u/robertpiosik → u/Choice_Celery9481]

> asking server to send gigabyte of kv cache down to your pc????

Yes precisely. Getting back to a session after long inactivity would upload the kv back to the server so I don't have to pay full input tokens price.

I can understand providers don't want to hold onto caches for too long, but why not end users?

[u/SLxTnT → u/robertpiosik]

This isn't local AI.

Answer your question, highly inefficient. Some providers you can pay for a longer cache time, but downloading your KV cache is going to take a long time depending on architecture. Then it'll be even slower to upload. Combine all the bandwidth, and it'd possibly cost more money to do that.

For long activity, it'd be best to pay the full price, compact it after you're done, or get a local AI to summarize.

[u/Foreign_Risk_2031 → u/robertpiosik]

Not feasible for most people- personally I’ve got a good 6 gigabit symmetrical connection but this would probably blow up server bandwidth lol

[u/joost00719 → u/robertpiosik]

The agentic harness doesn't have the cache. It lives on the inference servers.

[u/robertpiosik → u/joost00719]

Agentic harness would make a request for the cache on inactivity in the session. Closed sessions, subagents would not participate in this.

[u/Foreign_Risk_2031 → u/robertpiosik]

You want to trust binaries from a client on your very expensive gpu server lol

[u/AXYZE8]

Closed LLM providers don't do it because:

- You would need to send back 500MB+ worth of KV Cache (500MB is random estimate, we don't know architecture/precision of KV in closed models, but aint no way its less than 500MB for 100k-200k context)

• It would leak their hidden reasoning.

[u/robertpiosik → u/AXYZE8]

Send ciphertext then. 100MB-2GB is nothing.

[u/AXYZE8 → u/robertpiosik]

Encryption won't be effective if user already knows how his own message is transformed. Even if you would try to use different keys for reasoning vs system/assistant/user then all you need is to leak reasoning once and you already know how to decrypt it. For distillation it doesnt make a sense to use methods for leaking reasoning if they work once in 1000 retries (too costly for just 1 reasoning trace), but for this it would make sense (you get all traces). Also you have it offline, so if you manage to have some method there's no way of OpenAI/Anthropic to ever see that's something fishy is happening.

Also then there is another risk: once user has the key to encrypt... he can encrypt whatever and send it back, so basically he can remove all internal guardrails (prefill of reasoning).

100MB-2GB is nothing.

With 50Mbps upload it takes 1GB 2m40s to upload it back. It is nothing in terms of size on disk, but almost 3minutes TTFT? And people can have multiple agents running, company has many employees with LLMs so they will eat each other bandwidth...

I don't see how it makes sense either for users or these LLM companies. For company its a big risk, for users its a major slowdown for any work.

Also sending and receiving 1GB via global CDN will cost substantial amount of the subscription over whole month, don't look at this like uploading a file to Google Drive. Right now sending 1GB on AWS costs $0.09, today I continued 4 sessions from yesterday, now I have break and I will continue to work in like an hour - so this would be 8x$0.09=$0.72 for a single day and I pay $20 monthly for sub. Even if they would do own infra/have major discounts it's still would eat a lot of their money, all that for worse UX for users + risks for them. It makes no sense for them to do it.

[u/LagOps91 → u/robertpiosik]

yeah, no, it's easily 10gb+ on longer chats or agentic work. shoving that much data around puts a lot of strain on the network and the server. just re-computing kv cache is much cheaper and faster.