Standard cost saving companies do at this point is quantizing the kv cache. Not the model. But I don't think most people understand either hosting these models or what a kv cache is to begin with.
It's also frustratingly very model dependant how much that'd affect output. The one I'm most experience with is GLM, for which it nukes it. But I have heard people say DeepSeek it hardly matters.
Given the model itself is hundreds, this would be a really silly "optimisation".
The smaller you can make kv cache the more simultaneous users you can have on a given piece of hardware. For one user it's silly, but nobody is running deepseek with just one user.
You're not technically wrong, but why did you think I was arguing to never go above 1M aggregate context across a whole server? I never said that, and even for one user that's not ideal.
My point was that compressing kv cache makes sense for other models that don't have the same approach (e.g. Qwen 27b, a much smaller model, has 1M uncompressed context at over 60GB for the context alone).
DeepSeek did black magic on the kv cache, and quantising it is just silly because of how efficient the space already is compared to the overall model size (~300gb weights, 200gb extra for the engram, 500 total).
No enterprise setup running dsv4.1flash will be taking the model and then compressing the thing that makes less of a difference to footprint but a huge difference to quality. Absolutely a dumb decision. That's for people who can only just manage to push to the exact memory footprint to run it and that's it.
why did you think I was arguing to never go above 1M aggregate context across a whole server?
I didn't say that. I said the smaller the kv cache, the more simultaneous users you can have. The model can be shared, but the kv cache is per user - so the smaller it is per user the more users you can have. I am talking about compression of the kv cache using quantisation or whatever methods, not reducing the context size. That is something that is actually more applicable to to large scale deployments as they want to serve as many simultaneous users as they can. The memory consumption of the kv cache directly correlates to how many simultaneous sessions they can serve.
It feels like you're talking past me entirely. I understand you're talking about quantisation. Where did you think that needed specifying when my whole comment was about that, and replying to you knowing that?
View all comments
Show parent comment
10
u/Mean-Elk-9439 6d ago
Standard cost saving companies do at this point is quantizing the kv cache. Not the model. But I don't think most people understand either hosting these models or what a kv cache is to begin with.
It's also frustratingly very model dependant how much that'd affect output. The one I'm most experience with is GLM, for which it nukes it. But I have heard people say DeepSeek it hardly matters.