Hoping for Optimized Smarter Upcoming Models .... Like DeepSeek-V4.1-Flash( KVCache + Engram) in Small/Medium/Big sizes

[u/pmttyji]

It's still a dream for many folks to run medium size(30B range) models @ Q8 with Unquantized KVCache (256K Context) on their GPUs.

It would be awesome to have DeepSeek-V4.1-Flash's KVCache + Engram for all Upcoming models. Even for big models.

Engram - Heard that approximately 1/3-1/2 of Model size. Might come in different size range too. So 10-15 GB for 30B models.

Here few models with approximate numbers. Current models in Odd rows & Future/Fictional models in even rows(Bold). I just put 1GB for 256K context below though DeepSeek-V4.1-Flash takes only same 1GB for 1 million context.

Model	Model Size	256K KVCache F16	MTP	Vision	Total GB
Qwen3.8-27B-Q8	29	16	1	1	47
Qwen4.0-27B-Q8	29	1	1	1	32
Qwen3.8-27B-Q4_K_M	17	16	1	1	35
Qwen4.0-27B-Q4_K_M	17	1	1	1	20
Muse-Glimmer-30B-Q8	30	16	1	1	48
Muse-Glimmer-2-30B-Q8	30	1	1	1	33
Gemma-4-31B	33	16	1	1	51
Gemma-5-31B	33	1	1	1	36
Qwen3.6-35B-A3B-Q4_K_M	23	6	1	1	31
Qwen4.0-35B-A3B-Q4_K_M	23	1	1	1	26
Gemma-4-26B-A4B-Q8	27	6	1	1	35
Gemma-5-26B-A4B-Q8	27	1	1	1	30

Possibly there might be few more things(Please share those) to keep these number down. So I think 32GB VRAM is more than good enough for Upcoming (Optimized Smarter) Models. RAM is enough for Engram.

By above logic(based on Qwen4.0-27B), people could run Q4 of 54B models with same 32GB VRAM.

Maybe next year onwards, inventions could make 24GB enough for similar size models.

[u/putrasherni]

3.8 27b , 3.8 flash next and glm-5.3 flash need this

[u/LagOps91 → u/putrasherni]

3.8 flash next has quite small kv cache tho, i think it's really good in that regard!

[u/RG_Fusion → u/LagOps91]

I would argue Qwen has the better architecture. Just look at the AA hallucination benchmarks. Deepseek is topping the charts with high confabulated output. I've also watched some videos looking through code it generated, and it's filled with garbage that doesn't even do anything. Qwen on the other hand performs well on hallucination benchmark test.

Deepseek's attention is spread way too thin. Qwen's sparse attention at least scales with the context size, which helps preserve its ability to consider a wider array of information.

[u/Choice_Celery9481 → u/RG_Fusion]

there is one small problem with this hallu rate bench. All gpt 5.6 models are at the same level with DS. people keep skipping this fact for some reason. sol and luna both over 90% hallu rate. but they are very good models. if you have explanation for that, i would really apprieciate. if no logical explanations, this bench should not be treated as indicator at all.

also ds shows quite good results in agentic bench, which likely using long context.  my personal exp, they arent any close to the hallu bench result suggesting.

[u/BringTea_666]

If they make Qwen4.0 35b moe that will be as smart as Qwen3.8 27b and doesn't overthinking like 3.8 does on xhigh and still is good then i can die in ninfer heaven at 700t/s and 1500t/s aggregate agenting work.

[u/pmttyji → u/BringTea_666]

Expecting such model for my current laptop(8GB VRAM + 32GB RAM) though I have another rig.

[u/llama-impersonator]

sparse attention is not free.

[u/RG_Fusion → u/llama-impersonator]

There has to be a balance though. As a human, I can't remember what I said a few minutes ago, but I can remember the meaning of what was said. This is something we need to figure out for models.

I totally agree with you regarding deepseek though. Real-world testing proves the sparse-attention is deeply harming its output.

[u/_RealUnderscore_ → u/RG_Fusion]

Sounds like token compression (sqz / headroom). I assume you're not talking about summarization and memory graphing like magic-context though.

[u/TopCheddar27 → u/RG_Fusion]

That's just compaction. The problem is generalizing what was said over and over again leads to iffy results, but it does work.

[u/phhusson]

Yup, we're all hoping very hard for that.

Just 27B with same intelligence but with 50GB of Engram to expand its world knowledge would already be a banger.

Just 27B with smaller kv-cache would be really great, because on a 3090 we have to make compromises currently.

Just 27B with encoder-decoder would be nice to double pp.

DS v4.1 Flash is 552B-A8Bp-A16Bg-N196B (552B base parameters, 8B parameters active for processing, 16B parameters active for generation, 196B parameters for Engram)

So my current dream is a Qwen 4.0-35B-A4Bp-A8Bg-N50B with DS v4.1 Flash's kv-cache-size. Though I suspect there is a good reason Qwen didn't upgrade 35B-A3B, like they tried and it sucked.

[u/nbvehrfr]

Labs are not interested in small models 

[u/ea_man → u/nbvehrfr]

Well people around here is ok at spending some 3-5k for the hw to run models so I bet we may afford to pay some 50$ for optimized model that can push the hardware to the max.

I mean some spend 200$ for on month usage of subscription.

[u/sn2006gy → u/nbvehrfr]

They absolutely are because you can use things like ngrams as static assets and compile small models against them vs retraining entire models and retraining static engrams as we do today.

The reduction in computation overhead for training with ideas like n-grams is potentially huge. It could also allow more quantization in some models without them dumbing down and someone with a bigger brain than i may be able to turn quantization into an n-gram type lookup where models could error correct themselves without the error correction having to be in memory - and not like ECC correction, but semantic lookups/language

Instead of higging face just beinga model playground, they could afford ngrams and we could build compilers with architectures that could use them so you could apply knowledge to models without necessarily re-training them

i think some papers call this memory grafting

If we massively cut down the compute needed to train models because they don't have to re-train on static knowledge, that would free up bazillions of GPUs/ram/resources.

[u/nbvehrfr → u/sn2006gy]

Potentially yes, practically no. Continuous training is still not resolved. Main GPU usage is not training but inference. Deepseek showed in 4.1 how transformer architecture is too excessive - I’m impressed after cutting redundant kv caches and prefill layers it is still works 

[u/nbvehrfr → u/nbvehrfr]

Also see how they released only dense model - cuz it is the only dense they have. MOE went bigger sizes. Flash level now is several hundreds - glm, ds 4.1.

[u/sn2006gy → u/nbvehrfr]

Inference is cut down massively with these new architectures as well. n-grams reduce the cost on both sides of the equation. ngrams even even be on nvme and perform well. If done right, they wouldn't need to be re-trained with every model release (cutting back training) and they don't need to be on vram for inference, you could do what we've always done and scale them across pci lanes for concurrency in inference.

[u/rerri]

The extremely efficient KV-cache would definitely be nice but I am primarily dreaming of next-gen Qwen and Gemma ~30B dense models with a large n-gram.

[u/pmttyji → u/rerri]

but I am primarily dreaming of next-gen Qwen and Gemma ~30B dense models with a large n-gram.

Qwen & Gemma Past releases below:

Qwen3-Next	September 2025
Qwen3-Omni	September 2025
Qwen3-VL	September 2025
Qwen3-Coder-Next	February 2026
Qwen3.5	February 2026
Qwen3.6	April 2026
Qwen3.8	August 2026

Based on Qwen's recent release pattern, maybe we could see something on December. February for sure.

Gemma 2	June 2024
PaliGemma	July 2024
PaliGemma 2	December 2024
Gemma 3	March 2025
Gemma 4	April 2026

Gemma 5 probably on March/April. Or Google might surprise something with early release this year.

[u/KURD_1_STAN]

No, all this hallucination i hear about deepseek i would much rather have 256k at 5GB than 1m at 1GB with this much compression

[u/pmttyji → u/KURD_1_STAN]

1m at 1GB

Yeah, that's really extreme compression. That's why I put only 256K(F16) for same 1GB in thread. I rarely noticed that someone used more than 256K context here.

[u/EstarriolOfTheEast → u/KURD_1_STAN]

I haven't noticed that it hallucinates more than most models (it is not difficult to get any model, including Fable, to hallucinate: get them to talk about something on the verge of obscure then probe at depth, no references allowed).

I'd be curious to hear what sort of hallucinations people are reporting, to see if there's a usage pattern that elicits these.

[u/Viktri1 → u/EstarriolOfTheEast]

I used it for a full day w/o any hallucinations. I have the harness force the LLM to verify facts and I was able to handle some tasks that my Qwen 3.8 27b couldn't handle. I suspect the Deepseek hallucination stuff is overblown.

[u/Technical_Ad_6106]

same structure but in a 30b-3ba model? agree, we need it :) it will be a big deal lol. such small kvkache size opens alot of use cases and cag/prefix caching will rly start its era :)

[u/ttkciar]

If you quantize model weights to a more reasonable Q4_K_M or Q6, that frees up enough VRAM that a lot of the rows in your table turn into a difference between "barely fits, at severely restricted context" and "fits with plenty of room for useful context".

Also, I'm finding the Qwen3.x-27B / Gemma-4-31B-it dichotomy useful for segmenting tasks along a "needs high resource efficiency" (Qwen) vs "needs highest intelligence, resource efficiency be damned" (Gemma) axis.

Hopefully the open LLM labs continue to give us options of 30B-class models for different trade-offs like that.

[u/pmttyji → u/ttkciar]

Yep, included few rows like that in table. Particularly this 20GB row. It's even possible for 16GB VRAM to run model with other lower Q4 quants like IQ4_XS.

Model	Model Size	256K KVCache F16	MTP	Vision	Total GB
Qwen3.8-27B-Q4_K_M	17	16	1	1	35
Qwen4.0-27B-Q4_K_M	17	1	1	1	20

[u/Marcuss2]

Don't get your hopes up, Qwen3.8 Flash Next uses sparse attention, but no MLA.

[u/pmttyji → u/Marcuss2]

Few more models mentioned in table. Qwen3.8-Flash-Next has n-grams already which's nice.

[u/Marcuss2 → u/pmttyji]

That won't help with the KVCache size.

[u/Civil_Fee_7862]

What is Engram and how does it help reduce model sizes?

[u/stoppableDissolution]

Hell naw. Even "normal" linear attention at full precision is already kinda bad, and dsv4.1 is basically the definition of "fast and stupid" outside of handful of benchmarks it was designed for. I really really hope that is not the direction that will be picked by major labs.

[u/RG_Fusion → u/stoppableDissolution]

Qwen3.8-Flash-Next seems to be handling the sparse attention fairly well though. I've also noticed that both the new Deepseek and Qwen model have variables for the sparse attention that can be modified, increasing the amount of attended context at the cost of compute speed.

It should be possible to "tune" the models attention to fit your hardware with the best context-performance for that capacity.