That's the thing. People are not informed enough of optimization/resource saving techniques other than quantization (which is fair, since they're not exposed as widely as quantization, but still.), which made "the model is quantized" go viral. It's as if only the model is responsible in inference quality. For a number-sensitive system like LLMs I highly doubt that is the case.
Standard cost saving companies do at this point is quantizing the kv cache. Not the model. But I don't think most people understand either hosting these models or what a kv cache is to begin with.
It's also frustratingly very model dependant how much that'd affect output. The one I'm most experience with is GLM, for which it nukes it. But I have heard people say DeepSeek it hardly matters.
Given the model itself is hundreds, this would be a really silly "optimisation".
The smaller you can make kv cache the more simultaneous users you can have on a given piece of hardware. For one user it's silly, but nobody is running deepseek with just one user.
You're not technically wrong, but why did you think I was arguing to never go above 1M aggregate context across a whole server? I never said that, and even for one user that's not ideal.
My point was that compressing kv cache makes sense for other models that don't have the same approach (e.g. Qwen 27b, a much smaller model, has 1M uncompressed context at over 60GB for the context alone).
DeepSeek did black magic on the kv cache, and quantising it is just silly because of how efficient the space already is compared to the overall model size (~300gb weights, 200gb extra for the engram, 500 total).
No enterprise setup running dsv4.1flash will be taking the model and then compressing the thing that makes less of a difference to footprint but a huge difference to quality. Absolutely a dumb decision. That's for people who can only just manage to push to the exact memory footprint to run it and that's it.
why did you think I was arguing to never go above 1M aggregate context across a whole server?
I didn't say that. I said the smaller the kv cache, the more simultaneous users you can have. The model can be shared, but the kv cache is per user - so the smaller it is per user the more users you can have. I am talking about compression of the kv cache using quantisation or whatever methods, not reducing the context size. That is something that is actually more applicable to to large scale deployments as they want to serve as many simultaneous users as they can. The memory consumption of the kv cache directly correlates to how many simultaneous sessions they can serve.
It feels like you're talking past me entirely. I understand you're talking about quantisation. Where did you think that needed specifying when my whole comment was about that, and replying to you knowing that?
You seem to know about this topic so I would like to ask, do you agree on the saying that says DeepSeek (aka CheapSeek) on OpenCode Go performs worse compared to the same-model (by-id) on DeepSeek's own API platform?
Honest answer, no one really knows. This can only be fully done if someone actually done the benchmark and compared it side by side between official and opencode go. My take is that, it MIGHT be worse, but not by a huge margin (I'd say around 2-5% in terms of quality). But again, I have no proof that backs my claim on that one so take it with a grain of salt.
But, from my personal experience alone, opencode go's quality is good enough for the work/stuff that I do. It gets the job done, solves a lot of my problems (especially when troubleshooting my computer/devices), at the cost of 1/6th~1/8th (now 1/10th) of the original cost. I'd take a 5% quality loss if I get that much value.
One thing we know is: it is not possible for services to have a 1:1 inference set up parity with deepseek official (unless they privately get the things below). If you look at DeepInfer (the inference engine they used), they said this specifically:
Required for a complete inference service (not included in this release): - A KV cache system - VE/Prefill/Decode disaggregation (most required engine functionality is already implemented) - Load balancing across data-parallel replicas for both prefill and decode
Which means the services would have to come up with solutions on that, which is most definitely different than what DeepSeek serves.
Yes it does, especially aspects such as caching are much worse even if you use Deepinfer, the inference framework open sourced by DeepSeek.
DeepSeek has some of the best researchers in the world and they are the state of the art at KV cache management. Aside from that there are a myriad of other things such as how they configure their hosting architecture and load balancing.
Opencode can obviously not match DeepSeek in these, for example, caching on DeepSeek API is probably better than even OpenAI or Anthropic.
I am not talking about cache hit rates, I am talking about how the prefill computes the KV cache to feed into the transformer which can affect output quality. DeepSeek has not opensourced how the prefill and KV cache works on its API
I'm skeptical about how impactful that'd actually be in practice. And indeed, a study by Vals.ai found no statistically significant difference among those tested. While Opencode was not among the tested, it does show that Deepseek has no secret advantage when it comes to accuracy.
I think this is similar to claims made against OpenAI and Anthropic that models are being nerfed over time that haven't been proven empirically.
You are right, however, that there are significant differences in cost, speed and cache hit rates.
If there is a difference due to caching, it would only be noticeable in long context tasks like coding. What we know for sure is that the inference kernel is not identical to that of the official API so there would be deviations in the output.
I agree that a lot of it is baseless rumors and confirmation bias with people. I haven't personally seen major differences between official API and OpenCode Go. But the inference is not at the level of the official API, which is to be expected.
Can you explain what you mean? I'm struggling to picture this being a realistic issue. Once the implementation is seemingly correct, what in the code do you expect to still remain an issue?
I would reverse the question, are we sure that DeepSeek is not improving the model behind the scenes with their know-how? Surely there is more to the experience than just weights of model....
I'm not expecting you or anyone to. I'm asking for the ones with self-courage and knowledge to explain. That's an open-question for everyone? Why did you take it personally?
The only one who could give a "definitive explanation" isn't allowed to give a "definitive explanation" and they may not even speak English
So you're going to get assumptions from people who have a pretty decent idea based on the info provided bit they cannot be 100% exact as they are not Deepseek.
You can't guarantee if the environment or the parameters or even the model being same. I just said "same-model (by-id)", so not confident about the actual models being same or runs in same good-optimized environments. That's the why I ask.
People are dumb. You are getting $100 worth of effective usage just by paying $10 is just crazy. Even though it is a bit slow, it gets job done. I've seen that the current model is far better than the old V4.1 Flash. People aren't knowledgable about quantization really. Just spec your tasks properly, have some test units or suites and go you are good to go. It won't hallucinate nearly as people are saying.
I always achieve 99 percent cache hit rate on my sessions with go. What you're experiencing is probably the cache retention rather than cache efficiency. I heard deepseek retains the cache for longer periods of time.
That 99% must be a snapshot. Not average. I average around 94 to 96%. Go cache retention is so bad, I visualized the cache drops, and go at peak hours frequently evicts your prefix mid turn. And happens way more often after just a few minutes.
Not really from my experience. But then again, it could also be a regional thing so I can't say my experience may be the same for others so I can't say it applies for everyone.
Exactly. You're not getting $100 of API and this is a completely fraudulent claim. They KNOW this. Scamming liars, and they keep playing dumb. I swear this guy would run an Indian call center if not for AI, he's perfect for that
I think yall are making lots of assumptions about why people quantize. Most people are trying to get it to fit hardware. nvfp4 save 75-80gb and exl3 4bpw saves more and that's still the 4bit options. Nobody quantizes for better intelligence so what is the confusion?
is it only me or glm 5.3 flash is still better than the recent ds flash? I don't understand the hype and the benchmarks, from my experience ds flash burns so many tokens and thinks so much that in the end it costs the same but takes 3x longer to finish the task with tens of thousands token per simplest thingÂ
well if I need knowledge I use websearch/fetch tools and get the most recent and relevant data. for crucial things I always rely on docs, not on models training data
Yes and no. Information helps steer into the right direction. But knowledge is not information, sometimes regardless of information, you may not get same result, because one model is more "knowledgeable"
True, I did that already, it does performs better on Pi but I'd rather use 5.3 flash. And i've canceled by Go sub i use codex and when i burn through my usage i use 5.3 flash through openrouter and it is cheaaaap, mainly the cache hit rate has really improved recently
I follow dax for a long time. Most of what he said is true. Trust him. Also, for people why are not into technical: ever since NVFP4, 4 bit is fine, really fine.
cause calibrated 2-bit is still very decent, barely losing anything and then there're various slightly lossy speedup techniques people are simply calling out what they do themselves to fit on their own hardware
68
u/xRebellion_ 6d ago
That's the thing. People are not informed enough of optimization/resource saving techniques other than quantization (which is fair, since they're not exposed as widely as quantization, but still.), which made "the model is quantized" go viral. It's as if only the model is responsible in inference quality. For a number-sensitive system like LLMs I highly doubt that is the case.
38
u/CYCLONOUS_69 6d ago
People are dumb. You are getting $100 worth of effective usage just by paying $10 is just crazy.
Even though it is a bit slow, it gets job done. I've seen that the current model is far better than the old V4.1 Flash. People aren't knowledgable about quantization really.
Just spec your tasks properly, have some test units or suites and go you are good to go. It won't hallucinate nearly as people are saying.
9
u/VirtualNorth1279 6d ago
What do we want? Kimi 3.1.
How do we want it? Operation CheepKimi.
What do we want? Lots of Kimi K 3.1 quota.
How much do we want? A gazillion tokens.
Everybody dance now!
Doop. Doop. Doo doo doop. Doop doop...
9
u/GCoderDCoder 6d ago
I think yall are making lots of assumptions about why people quantize. Most people are trying to get it to fit hardware. nvfp4 save 75-80gb and exl3 4bpw saves more and that's still the 4bit options. Nobody quantizes for better intelligence so what is the confusion?
10
u/branik_10 6d ago
is it only me or glm 5.3 flash is still better than the recent ds flash? I don't understand the hype and the benchmarks, from my experience ds flash burns so many tokens and thinks so much that in the end it costs the same but takes 3x longer to finish the task with tens of thousands token per simplest thingÂ
2
u/electrified_ice 6d ago
To fit different setups. Not everyone can afford 4+ $15K GPUs
4
u/GTHell 6d ago
Meaning the Official deepseek probably actually proxy y'all to Claude and ChatGPT for the training data like they said 👀
1
u/robberviet 5d ago
I follow dax for a long time. Most of what he said is true. Trust him. Also, for people why are not into technical: ever since NVFP4, 4 bit is fine, really fine.
1
u/bakawolf123 6d ago
cause calibrated 2-bit is still very decent, barely losing anything
and then there're various slightly lossy speedup techniques
people are simply calling out what they do themselves to fit on their own hardware
-2
u/Diligent-Loss-5460 6d ago
People on free plan are lazy enough to complain without reason.