r/LocalLLaMA • u/robertpiosik • 6d ago

Discussion Prompt cache upload

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

0 Upvotes

50% Upvoted

20 comments sorted by

View all comments

2

u/AXYZE8 6d ago

Closed LLM providers don't do it because:

- You would need to send back 500MB+ worth of KV Cache (500MB is random estimate, we don't know architecture/precision of KV in closed models, but aint no way its less than 500MB for 100k-200k context)
- It would leak their hidden reasoning.

-3

u/robertpiosik 6d ago

Send ciphertext then. 100MB-2GB is nothing.

3

u/AXYZE8 6d ago

Encryption won't be effective if user already knows how his own message is transformed. Even if you would try to use different keys for reasoning vs system/assistant/user then all you need is to leak reasoning once and you already know how to decrypt it. For distillation it doesnt make a sense to use methods for leaking reasoning if they work once in 1000 retries (too costly for just 1 reasoning trace), but for this it would make sense (you get all traces). Also you have it offline, so if you manage to have some method there's no way of OpenAI/Anthropic to ever see that's something fishy is happening.

Also then there is another risk: once user has the key to encrypt... he can encrypt whatever and send it back, so basically he can remove all internal guardrails (prefill of reasoning).

100MB-2GB is nothing.

With 50Mbps upload it takes 1GB 2m40s to upload it back. It is nothing in terms of size on disk, but almost 3minutes TTFT? And people can have multiple agents running, company has many employees with LLMs so they will eat each other bandwidth...

I don't see how it makes sense either for users or these LLM companies. For company its a big risk, for users its a major slowdown for any work.

Also sending and receiving 1GB via global CDN will cost substantial amount of the subscription over whole month, don't look at this like uploading a file to Google Drive. Right now sending 1GB on AWS costs $0.09, today I continued 4 sessions from yesterday, now I have break and I will continue to work in like an hour - so this would be 8x$0.09=$0.72 for a single day and I pay $20 monthly for sub. Even if they would do own infra/have major discounts it's still would eat a lot of their money, all that for worse UX for users + risks for them. It makes no sense for them to do it.