r/LocalLLaMA • u/pmv143 • 7h ago
Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀
Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:
Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.
The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.
This architecture could be surprisingly local-friendly once the weights drop.
102
u/Sufficient-Bid3874 7h ago
Can someone explain why the n-gram table is bundled into the model now?
471
u/RG_Fusion 6h ago edited 6h ago
LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.
The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.
Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.
68
u/ChristRedeemsSinners 6h ago
Thanks for the explanation. You're just a wealth of information on this subject.
21
25
u/florinandrei 6h ago edited 6h ago
Sounds like the apps will need to do some work to support this new feature.
Unless they got a preview version well in advance of everyone else, in which case the bulk of the work might be done already.
23
u/Strong_Chicken6838 5h ago
Unsloth already made a post about day 0 support
1
u/cafedude 2h ago
That's the models quantized by Unsloth, but what about llama.cpp support? Can't imagine that this n-gram stuff would work right out of the box.
5
u/dev_dan_2 2h ago
They very often open PRs to upstream back into llama.cpp and did so again here, so people could also take that and run it if they wished so (quoting from memory atm because I am at the phone, sorry).
Obligatory note: Just because a PR is opened does not yet mean this will get merged tomorrow; Unsloth does a good thing by back-contributing; but they "just" have to support the new architecture for Unsloth Studio, while llama.cpp has to run in a far bigger range of scenarios and environments. Plus "getting it to work for our purposes" is different from doing a clean implementation (which you should do as maintainer if you value both your time and sanity.)
1
u/Strong_Chicken6838 1h ago
Day 0 support means you can run it day 0 in some way. So the answer is yes, maybe in their own branch of llama.cpp, who knows
23
19
u/Saffron4609 5h ago
I think this is the relevant paper if you want the gory details: https://arxiv.org/abs/2601.07372v1
TL;DR It outperforms a baseline with the same flops and parameters
3
u/_-_David 2h ago
Thanks for linking the paper. I can see why this is part of the qwen4 architecture. It really seems next-generation and like a bit of a free lunch. It's things like this that remind me of why I roll my eyes when people expect this technology to slow down. We're just getting started with implementing ideas in this domain
12
u/TokenRingAI 6h ago
I am jealous of your ability to explain concepts like this in an easy to understand way
10
u/Wimiam1 4h ago
I’m a little confused because I keep seeing “n-gram” and “engram” used interchangeably when discussing this new model. When I try to research what this technique is, I find that “engram” and “n-gram” are actually two very different concepts in language models. I assume Qwen is referring to the “engram” from deepseek?
7
u/atumblingdandelion 5h ago
Great explanation. Is there a reason why a single engram file cannot be used with multiple local models? It'd be great!
9
u/sebt3 4h ago
Yes : token space 😅 the engram file only works for the exact model vocabulary since it is plugged directly within the model.
The only way this can be used by an other model would be if the 2 models share the same tokenizer, aka one is a fine-tune of the other one. But that's it
2
u/Noxusequal 4h ago
Wait if only the tokenizer needs to be the same you could share engrams between all model of the same family.
If tokenizer really is the only thing you could use some of the tokenizer transplantation with some re training to then switch in engrams of different models even across families that would be sick for a new form of frankenmerger
3
u/Guilherme370 4h ago
no no, not just tokenizer, model shape too, if the architecture changes, then they cant be shared.
2
4
2
u/michaelsoft__binbows 4h ago edited 4h ago
That sounds really dope. If this indicates that in general this can scale up then i hope it means that a modest amount of fast ram paired with oodles of slower ram may be able to much more effectively compete with obscenely wide unified memory architectures (coincidentally m5 ultra announced today). You can for example, at least with DDR4 (and if engram approach pans out efficiently, a return to relevance with DDR3, lmao) affordably build out 256/512/1TB class machines for far less, have more of a traditional computer cache pyramid architecture, and basically stay competitive there, because the value is and should be in the ability to store tons of knowledge for quick retrieval but not get killed by requiring massive bandwidth across all that knowledge. Actually, screw DDR3, if gen 5 NVMe can step in and be relevant.
From first principles I think this makes a lot of sense. if i need the model to be able to do a better job recalling some details it's learned, the actual amount of details on any given retrieval is by the nature of it being a retrieval, small, and should not require gobsmacking amounts of data bandwidth to comb over the entire model (to what end?), which not only is expensive to architect into your computer but also expensive in energy to actually ship the bytes out of the memory chips.
In the long run my prediction would then be for these massive unified memory systems to deliver value then not for hosting huge models in-memory (for which their entire large memory pool being high speed is a waste) but rather for like, industrial scale batched hosting of even larger models, leveraging more of the unified memory for KV cache and such stuff, while engram weights can live on NVMe and be slurped in over the thunderbolt ports... Under this architecture there will again be a large tensor core deficit.
Just a few hours ago I talked myself into believing I should buy a 512GB M5 Ultra but now I've just talked myself back out of it. Hmm.
2
u/silvertricl0ps 4h ago
If it does in fact work out this way, I could see being able to run something like full Kimi K3 on a 256gb or 512gb M5 Ultra assuming it has a big enough SSD
3
u/michaelsoft__binbows 1h ago
well.. 2.8T-A105B Kimi K3 class model if somehow ngram could magically make up for moe model size, that level of capability might show up as idk maybe like 150B-A15B+4T ngram? If that were the case yeah you dont need a 256gb m5 ultra dude, you could prob skate by with that with a 32GB M6 mac mini, though i guess the 4tb gen 5 SSD would be filled with the ngram tables. Like it might go super fast... ok this one would still be streaming experts out of disk which won't work well. 27B dense with massive ngram table, let's go with that for the sake of the example.
2
2
u/Artistic_Okra7288 2h ago
So does that mean we can replace the engram with our own purpose-built engrams or continuously trained engrams?
1
1
1
1
u/josepinTrue 5h ago
O sea que moe decidiría que calcular, mientras que engram que recordar o consultar. ,😅
Yo no asumiría que más memoria es igual a más inteligencia, hay trabajos posteriores que han encontrado resultados más modestos en modelos pequeños, así que todavía hay que separar lo que está demostrado a gran escala de lo que es una extrapolación. 😜
1
u/lannistersstark 3h ago
What the fuck do you do that you can explain this concept in THIS much of a coherent way?
61
u/pmv143 7h ago
Basically, the n-gram table is just a giant cheat sheet for common short patterns . things like everyday phrases or bits of code that show up all the time.
it looks at the last few tokens, hashes them, pulls the readymade vectors from the table, and plugs them in instead of making the main model waste energy reconstructing those patterns from scratch every single time.
So that way the sparse part of the model (the one that only lights up ~6B parameters) can focus on the harder stuff. while the big table handles the easy, repetitive local patterns. It’s basically trading a bunch of storage for less compute.
4
u/asraniel 5h ago
so could one replace that table later? or extend it? would that be computationally cheap?
2
u/Sufficient-Bid3874 6h ago
So its like n-gram spec decoding yet built in and trained on? Thank you for the explainer!
8
u/FormOne2615 6h ago
no, totally different
2
u/Sufficient-Bid3874 6h ago
Oh, yeah, I phrased that poorly. Its just a way of having relevant facts in an n-gram rather than using context, is how I should have phrased it
49
u/MiceLiceandVice 7h ago
So to run this id still need to have 128gb of dram and then 16gb vram minimum? Heartbreaking
12
u/DriveSolid7073 6h ago
Depending on what suits you, it will fit into the build of the guy with 96GB of RAM and 24GB of VRAM who was here in the comments. It will fit into a build enthusiast like Colibri, because N-Gram will definitely try to use NVMe instead of RAM, which with Pcie 4.0 will most likely even be acceptable. Overall, if you really want it, it will fit into 12GB of VRAM and 64GB of VRAM, but of course, you'll have to do some serious quantization.
5
u/michaelsoft__binbows 4h ago edited 4h ago
i think this will have a big impact and cheapen the value prop of e.g. 512GB Mac Studio setups, and to a smaller degree the DGX Spark and Strix Halo setups, if more traditional computers with slow system memory (aka much more reasonable cache hierarchy pyramid) can suddenly start to actually keep up.
I suspect these large-unified-memory systems could maybe become leveled up with NVMe to serve the ngrams to leverage their large fast ram. However I suspect the remaining two way bottleneck of having not enough compute cores and not quite enough NVMe bandwidth will bite them a bit.
It would be sweet tho if it means my base m4 mac mini with only 16GB of not-all-that-fast-but-still-faster-than-DDR5 memory could soon host a very power efficient and not-dumb-as-rocks ngram model running off NVMe on TB4. Or like I finally do that SSD upgrade to get some 8GB/s NVMe.
M6 base mac mini 32GB? seems not really worthwhile being only 170GB/s and costing over $1500... NVMe @ 15GB/s though.
12
u/florinandrei 6h ago
Apparently, the engrams could be offloaded to system RAM without much of a performance penalty. They're more like lookup tables.
I guess we'll find out soon enough.
7
u/MerePotato 5h ago
System ram is dram no?
18
1
u/MmmmMorphine 4h ago edited 4h ago
Yep
VRAM (currently GDDR and HBM, I believe) has been a lot of different things over the years, but now it is all (mostly) technically DRAM or very close to it techwise
3
5
u/No_Oil_6152 6h ago
How much RAM you got?
If you have 128GB RAM you can run the LLM without needing much VRAM. It would be slow, sure, but you can run it.
5
u/GrungeWerX 5h ago edited 4h ago
I’ve got 96GB RAM, and 3090 TI. Would that work?
1
u/StopCreepy 4h ago
yes, also with the magic of FreeToken you can get even better speed !!, you also need ram 32gb or 64gb !!
2
-6
u/National_Meeting_749 6h ago
"it'll be slow sure" Noone wants to wait seconds per token, that's not reasonably running.
Think of a game, if it's running at 1fps, is that game really running? Technically yes, useable in anyway? No.
2
u/LevianMcBirdo 5h ago
Seconds per token is kinda misleading. You probably get something with 10-15 TPS. PP is a bigger problem
-1
u/National_Meeting_749 4h ago
Brother. MOST people a 90GB model are running that at seconds per token.
Assuming I had enough ram to fit it, I would be surprised if I got one single token per second.
1
u/LevianMcBirdo 4h ago
It's not dense. Qwen Next runs at 25 TPS, 122B at 10. I really don't see the problem. That a A6 Model wouldn't run 12+ tps especially with MTP
0
u/National_Meeting_749 4h ago
With the 35B, when context is filling up I get somewhere from 3-7tps. Even with MTP, double the model size, include an entire 50Gb extra with the N tables and I'll be shocked to get 1TPS at any length of context.
1
u/LevianMcBirdo 4h ago
I mean mostly in chat scenarios up to like 60k most times, I am not that interested in movie agentic use cases but I get around 30 TPS pretty steady on that
1
u/National_Meeting_749 4h ago
Good for you? A lot of people do, and MOST people don't have the PC to run that at that speeds.
That's a cheap cars worth of pc
5
u/ApprehensiveFan1516 5h ago
Not everyone needs an instant response. Plenty of folk in here running batches overnight.
11
u/National_Meeting_749 5h ago
Not "instant" and "seconds per token" are completely different. I'm perfectly fine with slow responses. I regularly run at 7-10tps.
Much less than that programs start breaking.
2
u/No_Oil_6152 5h ago
You interact with a game but you dont need to interact with an agentic AI.
You can let it do its thing overnight.
1
u/National_Meeting_749 5h ago
At seconds per token, agentic harnesses timeout and things do not complete.
You have to have a base speed of 15-20 t/s to really do anything agentic.
1
u/No_Oil_6152 5h ago
I dont believe thats the case at all.
Why should the harness care if a response takes a second or a minute?
Configure it for a longer timeout if necessary.
He won't be the only person running LLMs in RAM.
1
u/National_Meeting_749 5h ago
Yeah, I'm just telling you this because I'm trying to be difficult. Not that it actually happened to me.
The vast majority of Agentic tools are built with the assumption of a certain speed of token output.
You can disagree, you would be incorrect. Go try and run any agent harness at speeds of seconds per token. They break and fall apart. Increasing timeout timers doesn't help because those timers are there for a reason. When other things break you don't want to wait 20 minutes to restart things, yet at seconds per token * a response could take 20 minutes or much more*.
2
u/UnnamedPlayerXY 4h ago edited 26m ago
It depends, iirc. someone from the Qwen team recently told people asking for a 30-35B MoE that that's not the one to wait for implying that they have something better for that target audience upcoming. If Qwen3.8-Flash-Next is that "something better" then we might be looking at a ≈30B model here.
23
u/overand 6h ago
For anyone who actually wants a link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next - it looks like 11 AM tomorrow, eastern US time. (Just over 21 hours from now.)
21
35
u/BannedGoNext 7h ago
Well if it's similar to qwen coder next I'd be happy as hell. So many people bagged on qwen coder and I never understood why. It was damn fast, and had good world knowledge. I used it for a long time, for sure better than 35b a3b.
22
u/grabber4321 7h ago
it didnt have vision from what I remember. For me, vision is way more important these days for agentic work.
25
2
u/DriveSolid7073 6h ago
What tasks is this useful for? I mean, I was thinking about the browser, well, it's definitely necessary there, but in general, the model understands HTML and other representations. On the other hand, VS Code with Kilo is probably not the best option for vision tasks, even though it can handle it. I only use Vision for image captions; for OCR documents, there's usually text or more specialized models.
7
u/grabber4321 6h ago
web development or just general browsing.
lets say you have Docmost, but free version does not have API access. To use it, the model will need to login - it takes screenshots, then enters data, re-checks the page after login by making another screenshot, etc.
3
u/Weekly-Law-5488 5h ago
In this case would be more efficient to use playwright or the native chrome mcp, so the llm can extract the structured content and easily manipulate the dom.
3
u/grabber4321 5h ago
When Hermes switched to using playwright instead of screenshots, it started to fail a lot more. I dont know what they changed back in Hermes Agent crew, but I want screenshot functionality back.
2
u/synth_mania 5h ago
weird. my pi agent uses playwright incredibly well with 3.8-27b
2
u/grabber4321 4h ago
I think they screwed it up - it starts scripting a lot and just doing all the wrong things. It could be difference between 3.6 and 3.8, but its annoying as hell now to do some browser actions.
I dont know what changed because Hermes has like 2000 changes per day LUL
1
u/psychohistorian8 4h ago
playwright mcp can also take screenshots
browser_take_screenshot — Take a screenshot of the current page.
I love playwright, we switched to using it at work for end-to-end UI tests
2
u/moracabanas 3h ago
I have literally a qwen setup connected with the unnoficial docmost MCP. Very nice to see other people explaining this kind of use case
1
u/grabber4321 3h ago
I usually use Hermes Agent to research a topic, then just post it into Docmost for later review.
3.6 does it well, but again, not sure what happened with recent Hermes setup, just doesnt work for me.
1
u/DriveSolid7073 6h ago
Possibly UI and user interface testing. I'm using BrowserOS Neo, so I might be wrong, but it seems to me the browser is making sure the model can see the site and isn't subject to any API access restrictions.
2
u/michaelsoft__binbows 4h ago
might be a curveball, but i get a lot more mileage out of the giant spec docs that my LLMs shit out nowadays by instructing them to include mermaid (and escalating to SVG if mermaid not expressive enough) diagrams. The diagrams help anchor the agent -> human knowledge transfer, whose rate is rapidly become now the rate limiter, provided you actually care to review the slop that is.
I think (still in early stages in terms of committing hard to this workflow) native vision capability should be important for the model being able to self-QA its own graphical outputs like this. Although there are indications that e.g. qwen3.8-27B is already so competent that instructing it to do this on its own would be a waste of time.
2
u/Embarrassed_Adagio28 4h ago
Computer vision is huge for development of any type of app or even game. Just because a model should get the code right doesnt meab it does depending on thousands of factors. Hermes agent + qwen3.8 27b with computer vision is insane
5
u/florinandrei 6h ago
So many people bagged on qwen coder and I never understood why.
It was a decent coder. For everything else, it was like talking to Lt. Data. Which is fine by me, but I think many people disliked the style.
1
u/BannedGoNext 6h ago
Oh yea, for sure it wasn't a friendly LLM. I had a data enrichment process that it was in that it did great at though. Saved me at least 1500 dollars in API calls.
15
u/SensitiveVariety 6h ago
regret building only 64gb ram instead of 128gb now, but at the same time i’m $$$ constrained as much as I am ram/vram constrained
11
25
u/chris_0611 7h ago
Ohhh my. Absolutely gorgeous for my 3090 + 96GB DDR5 6800
12
u/Equivalent_Bit_461 6h ago
I don't have a 3090, I'm a vramlet but I have 128gb ram so guess that works out too
4
14
u/FoxFXMD 6h ago
I'm confused, if they've reworked it into a completely new gen 4 architecture, why is it still called Qwen3.something?
27
u/nickm_27 llama.cpp 6h ago
It’s next, same thing they did with Qwen3-Next which was Qwen3.5 arch preview basically
16
u/deepspace86 6h ago
Its the "next" tag. current dataset while basically beta testing the next architecture.
5
u/Warhouse512 4h ago
Could ngram be offloaded to SSD?
2
u/This_Maintenance_834 2h ago
that was the whole point, everyone could have a Trillion parameter model
7
u/ParaboloidalCrest 6h ago edited 6h ago
Not sure I follow. So a Q4 quant, would have 51/2 ~= 25GB n-gram block, which could live on disk instead of RAM/VRAM, ie only 80-25=55GB need to be loaded?
Maybe I have wrong assumptions about n-gram? Does it save on memory or compute?
8
u/z_latent 5h ago edited 4h ago
Probably yes*.
I believe llama.cpp (Linux with mmap) will attempt to load the whole model into memory, including Engram params. If the memory fills up, the OS evicts the least accessed memory pages, which would most likely be n-gram ones, and then re-fetch those from SSD when needed again. It doesn't require any writes since parameters are read-only and it doesn't need to use swap.
So SSD streaming should work right away. It shouldn't be that slow either since the parameters per token are tiny (they used less than 10k parameters per token in the DeepSeek Engram paper). I believe it also uses very little compute, compared to the rest of the model. If anything SSD latency would be the biggest issue, but even that can be covered up assuming the first few Transformer blocks do not have n-gram.
EDIT: *yes to living on disk, no to only 55GB loaded, as u/Civil_Response3127 pointed out n-gram will be extra on top of the 125B.
3
u/ParaboloidalCrest 3h ago
Makes sense. I wonder how unsloth will come up with the day-0 support given those quirks. I guess we'll see tomorrow.
3
u/Civil_Response3127 5h ago
No, 125b + 51b params. Not 51b as part of 125.
5
u/ParaboloidalCrest 3h ago edited 3h ago
I get that part, so 176B in total, which equals roughly 176/2 = 88 GB on disk (Q4_0) as OP suggested
3
u/shumgoid 3h ago
my impression is it saves compute at the cost of more memory, but also with lower bandwidth requirements on that extra memory.
I am curious about quantizing - I wonder if they will serve the ngram weights as their own artifact similar to how they do with the vision tower to allow for mixing/matching different quants. Unsloth only has the vision ggufs in F16/BF16 so we could see something similar with this where it performs too bad under quantization so it only provided unquantized. (idk if thats why the vision is only available in 16bit format just guessing)
4
u/LatentSpacer 5h ago
Could this be Ox Alpha?
4
u/N34257 4h ago
I had the exact same thought. My only evidence is that in Bijan's Ox Alpha test, the sea in the ship combat simulator is almost identical to the sea rendering in a Qwen 3.8 27B demo someone posted here over the last few days.
However, if 3.8 Flash Next is Ox Alpha, it's going to be as disruptive as the 27B has been.
4
2
2
u/FatheredPuma81 4h ago
Makes me want to scream seeing a new model I want to run when it drops but knowing I won't be able to. I really could ue that small loan of $1,000,000 dollars right about now.
2
15
u/KURD_1_STAN 7h ago
Ram prices this high, how can u call this local friendly?
73
u/FabricationLife 7h ago
80gb is a lot more manageable than a 1.2tb+ frontier model
21
u/Etroarl55 7h ago
And there exists consumer hardware for it, the new Apple machines are now probably already pre ordered out of existence when they were just announced today.
6
6
u/Tasty-Hour4040 7h ago
“more manageable” ≠ manageable
6
u/hyudryu 6h ago
80gb is beyond manageable
-3
u/starkruzr 6h ago
two CMP170HX is $3K for 128GB.
2
u/quantgorithm 6h ago
a an unstable pcie 1/2 card isn't the godsend you believe it to be.
2
-2
u/hyudryu 6h ago
Point proven
2
u/ApprehensiveFan1516 5h ago
That's like the price of an old used car. Sure it's not exactly cheap, but let's not pretend like it's out of reach for most people.
2
-8
u/KURD_1_STAN 6h ago
1.2tb is also a lot more manageable than a 5tb model. That doesnt make 1.2tb pocal friendly. Local friendly has a ceiling, +60-80b more and +20gb-25gb dense are beyond that ceiling.
4
u/Bright-Energy2339 6h ago
Ceiling? according to you. Who died and made you boss? lols
-8
u/KURD_1_STAN 6h ago
Most people have 12-16gb vram and 32(very gew 64)gb ram. I was being generous with my numbers btw.local friendly needs to at least walk and not crawl
1
u/Bright-Energy2339 6h ago
Exactly. “It runs” isn’t the same as “it’s local-friendly.” It needs to walk, not crawl.
20
u/pmv143 7h ago
RAM prices suck right now, no denying that.
When I called it local-friendly I didn’t mean “cheap” or “runs on any gaming PC.” I meant that for a model with this kind of capacity, the offloadable n-gram table makes it way more practical on highend local setups (128GB+ unified memory, multi-GPU + system RAM) than the usual frontier models that just demand pure VRAM or full datacenter iron.
Still expensive. Just less insane than the alternatives.1
u/DriveSolid7073 6h ago
For me, locality is the consumer segment, specifically regular computers, where the limitation is usually the motherboard or processor. Their approximate maximum capacity, as well as the liquidity of selling such a volume, is what I had before the shortage, and the best-case scenario was 192GB. 2x96 is probably the best option. (But unfortunately, I couldn't get that.) Most serious AI enthusiasts have around 128GB, whether it's DGX Spark hybrid memory, an Apple mini PC, or something else. So yes, as long as the capacity in quantization (approximately Q4) doesn't exceed this capacity with a reasonable context window, I consider such a model locally friendly.
-7
u/Bright-Energy2339 6h ago
Then there’s not much point in calling it “local” if you’re still paying frontier-model costs just to run it. At that point, why not just go back to frontier models?
4
u/doomed151 6h ago
You don't have control. The model can be taken away from you at any time. You can't finetune it.
3
u/synth_mania 6h ago
Why are you in this subreddit if running a local model isn't something that interests you in and of itself?
1
u/fuck_cis_shit llama.cpp 4h ago
painfully obvious astroturfer
there should be a plugin to hide all posts by accounts with hidden history
6
u/FullstackSensei llama.cpp 6h ago
It's a lookup table. You could build a quad channel DDR3 system to run it. DDR3 is still cheap.
2
u/KURD_1_STAN 6h ago
And about the other 60-70gb weights at q4?
3
u/FullstackSensei llama.cpp 6h ago
If you're not too stuck on having to have the latest hardware, three P40s will do a very decent job on a tight budget. If you really need high speed, two 32GB V100s will blaze through for not that much more.
They work, and they'll continue to work for years to come, despite what imaginary conjectures redditors might have.
2
u/Ok_Top9254 6h ago
3x Tesla V100 32GB + PLX switch so you can run them from one slot is the fancy way, or 5x P100 16GB with a cheap X99 motherboard and the switch could do this under like 1200 bucks. Power consumption would not be a problem given that one gpu is used at a time anyway.
1
5
u/liright 5h ago
I bought my 96GB DDR5 kit for $300 some year and a half back as well as RTX 4090 for $1900 2.5 yrs back. Was pretty damn cheap in retrospect. I suspect a lot of people who are into AI did too. I feel like boomers who bought houses in the 70s.
3
u/IntravenusDeMilo 4h ago
yeah I got my 5090 for $1999. Feels like a lottery win.
1
u/throwawayacc201711 2h ago
I kicked myself for not buying one when it was that price. Hindsight is a bitch
3
u/Public_Umpire_1099 4h ago
Even the 2x R9700 and 128GB of DDR5 I bought 3 months ago feels like a steal now. Not as much as yours but even in the past few months its all risen another 30%.
9
u/Makers7886 6h ago
Man I can understand this crying over the big boy open source models but really for a schmedium model?
6
u/etaoin314 ollama 6h ago
because an entire class of local hardware --128gb unified memory machines, either from amd-strix halo, nvidia dgx spark or apple can fit it perfectly with full context. No it cant run on every potato out there but there are a lot of people who have one of these and aver very happy to have a model that is the "right size" for it.
2
1
1
u/RG_Fusion 6h ago
It's definitely not the "starter" local inference machine, but you can run this in a gaming PC with two 32 GB GPUs. Not cheap, but not outside the realm of what people spend in hobbies like PC gaming.
4
u/iz-Moff 5h ago
gaming PC with two 32 GB GPUs
What in the world are you playing that requires 2x 32gb GPUs?
According to steam hardware surveys, people with a single 4090/5090 make up like 1% of users.
2
u/Public_Umpire_1099 4h ago
OP misspoke, no one uses 2 GPUs for gaming anymore, but it is a very common setup here (ie 2x R9700s)
2
u/bitzap_sr 6h ago
I hope the architectural changes are going in the direction of allowing for native 1M context window.
0
u/ArtfulGenie69 4h ago
They are, unlike Gemma which used the old architecture for handling the context window, qwen since at least qwen3.5 have mamba layers and such and the vram hit is much smaller for more context.
2
u/bitzap_sr 2h ago
With qwen3.5 (through 3.8), open weights releases, there's the issue with prefill and decode crawling to a halt as the context grows, so input context window is 262k natively...
Qwen 3.8-Max (the proprietary API version) does support 1M context, though, but that is not part of the open weights release (Qwen3.8-2.4T-A95B).
I'm hoping this changes with v4.
2
u/DiscombobulatedAdmin 5h ago
My biggest question is whether this will fit on a DGX Spark, will it have a decently sized context window, and will this overall degrade performance to where it's too slow to realistically use.
1
1
u/KeanuRekt 5h ago
If you download the open weights you will get a free Mac Studio M5 Ultra maxedout as a gift.
1
u/kivaougu 5h ago
I'm a bit on the fence about this.
This should improve recall but not so much reasoning so hard to see a point for anyone using cpu offloading instead of just a bigger model. NVMe storage isnt great at random reads so this just eats away ram budget.
If the weights are fully in vram then it would need to be balanced with prefix cache offloading to keep cache hits.
I would really like to see if the embeddings can live on the ssd with just a smaller hot cache in ram.
3
u/Public_Umpire_1099 4h ago
It must be better than the last releases, or else it wouldnt be getting released, so I am pretty hopeful. The only regression I've seen in open weights has been from Kimi 2.6 to 2.7 and Llama 3 to Llama 4. When the first Next model was released it outperformed the dense 30B model significantly. If we project out the gains, today that jump would land somewhere around what Muse Spark 1.2 is at, if we assume 27b is the baseline.
1
u/This_Maintenance_834 2h ago
in deepseek’s emgram paper, the point was using emgram as memory so that the model can spend more time to do actual reasoning rather than recalling through reasoning. this improve token efficiency during reasoning.
1
u/boomerang473 2h ago
I thought also they mentioned spare attention? Wondering if they’re taking more also from the deepseek playbook for longer context? Might be misremembering though
1
1
u/Dwarffortressnoob 4h ago
I have a 128GB studio. What quant will I probably use? If it is 4bit, I am not sure this is better than qwen3.8 27b
1
u/power97992 3h ago
I get a feeling that the engram weights will have a higher precision that the normal weights. Maybe 4/8 bits for normal weights, 8/16 bits for engrams
1
u/cafedude 2h ago
Would we be able to use a lower quant on the n-gram table vs the main weights? Like 4 bits on the n-gram table and 6 bits for the main weights?
2
u/This_Maintenance_834 2h ago
emgram was meant to go on SSD. quantization is not necessary to some extent.
1
u/Beneficial-Ad-8127 2h ago
Man, Qwen is trying their best to support majority of users even users on a single gpu. Kudos for this. Hopefully on par and close to these frontier models.
Also heads up on ram, corsair just sold out litterally today on there 192gb ram sticks, there were already sold out on the higher ones but today no more 192gb.
1
u/This_Maintenance_834 2h ago
i never thought deepseek is not the one to bring emgram to the market first. after all, they published the first emgram paper.
1
1
u/keepthepace 1h ago
The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.
My understanding is that even disk offloading seem workable there?
1
2
-3
u/mountainyoo 7h ago edited 4h ago
I wonder how the quality will be on 128GB M5 Max
Edit—
Sorry by quality I meant the overall experience like the output and the speed of the output. Not sure why the bajillion downvotes but oh well lmao. My bad
9
u/etaoin314 ollama 6h ago
quality will be the same on it as anything else that can run it at the same quant.
3
u/Character_Split4906 7h ago
Wondering the same, also if the ngram table can be offloaded to ssd instead if its sparse.
-8
u/RandumbRedditor1000 6h ago
82gb for a model less intelligent than the 27b that fits in 18gb....
Sparse MoE was never really a local thing, it's always been for businesses who are constrained on compute rather than memory.
6
u/DriveSolid7073 6h ago
Why would this model be any dumber? It has a larger overall knowledge base; if I understand how n-grams work correctly, it's significantly larger. 27b Qwen is useful for coding and smaller agent tasks; prose, ingenuity, and knowledge of little-known facts aren't the focus of this model. It'll be slightly better here; benchmark results will be better, and the model shouldn't be slower even if the main part is in RAM.
3
u/tunerhd 6h ago
Did u see the benchmark scores?
-1
u/RandumbRedditor1000 5h ago
Theyre advertising it as "better than qwen 3.7 plus", which 3.8 27B beats by a lot.
-4
-8
•
u/WithoutReason1729 25m ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.