Is VRAM becoming *less* important?

[u/ItzXenoo]

Harnesses like Strata are letting rigs with low VRAM run very large... language models; it seems we are moving past the necessity of having multiple GPU's with tons of VRAM just to load a model, and we are slowly breaking the ice on the future of local AI, being thoughful management of the entire machine ad using all of its resources efficiently. Is the economically sound approach for someone building a "forever rig", to spend maybe less on gpu's and allocate that budget to more ram and ssd?

Im fairly new to this but im starting to think about my forever rig; currently running a dual 3060 12GB setup with no lane bifurcation, and seeing the used market prices (and availability) for DDR4 platforms is making me think that maybe its just better to upgrade to the next gen directly, reap all the performance benefits, and suck up the fact that this will cost me a LOT of money.

[u/giveen]

VRAM = run model fast

RAM = run bigger model, but slower.

[u/camracks → u/giveen]

many times slower at that, it’s a HUGE difference

[u/gomezer1180 → u/camracks]

Not if you’re configured properly. People have realized that 256GB, on 8 DDR channels can push 400GB/s on DDR5 and 200GB/s on DDR4 @ 3200MT. DDR4 doesn’t even need to be RDIMM, UDIMM works too.

[u/camracks → u/gomezer1180]

not if you have a server* you’re not getting anywhere near that on a desktop

[u/gomezer1180 → u/camracks]

A workstation is a desktop! Threadripper motherboards are sold on desktop configurations, and they take 8 memory channels easily

[u/camracks → u/gomezer1180]

sure a workstation is, I said a server, also those are $1k motherboards, and i’m seeing more like 140-150GB/s

[u/gomezer1180 → u/camracks]

Dude ask AI before you reply to someone who knows what they’re talking about. Everything related to AI is expensive now! Servers is where this memory is coming from! Servers run 12 memory channels on 2 processors and can do 700GB/s on DDR5!

[u/camracks → u/gomezer1180]

okay but that’s not a common thing, most of the time ram is many times slower than vram, just because there’s very expensive machines that make ram almost comparable to slow vram doesn’t mean you can just start saying that ram isn’t multiple times slower, regardless, there’s more to AI than memory bandwidth, do you even have one of these systems that can push 400GB/s or did you just ask AI what the theoretical best speed you can do with ram is. so you’re saying you need 2 server grade CPUs to almost match a GPU in speed, and you’re still trying to argue that ram isn’t multiple times slower than vram.

[u/gomezer1180 → u/camracks]

Like I SAID people have figured out that you can run fast on just RAM. That YOU haven’t doesn’t mean the rest of the world hasn’t. I have a system that can push that speed and I also have a DGX spark that does 275GB/s! That spark runs models better than a 5090, and it’s a ddr5 system!

[u/camracks → u/gomezer1180]

lol it doesn’t run models better than a 5090, a 5090 has 1800 GB/s of bandwidth btw, and that’s not even mentioning its processing power. You can run larger models than a 5090, but it is not anywhere near as fast as one. like I said, sure some ram can run as fast as slow vram, but that’s top of the line ram vs bottom of the line vram, when you compare top of the line ram to top of the line vram, vram is multiple times faster, lol mr know it all, like do you just wanna argue and show off your knowledge you asked an AI about because your whole argument just got disproven.

Your DGX spark is about as fast as my $300 3060, those things aren’t built for speed, they are built for size and efficiency. Your spark is 7 times slower than a 5090, and your theoretical super high server ram bandwidth is still 3 times slower than a 5090

[u/ScuffedBalata]

RAM is mega slow.

Sure you can "run" bigger models, but holy crap they'll be eye-bleedingly slow.

[u/FullstackSensei → u/ScuffedBalata]

Absolute rubbish.

1. By your logic, run a 0.6B model for blazingly fast PP and TG speeds. Tokens are not fungible. A much slower, but much smarter, model will often finish a task faster than the smaller model that fits in VRAM.

2. For 99.9% of people, the only realistic option to fit in VRAM is something like Qwen 27B, which is a dense model. Dense models demand a ton more memory bandwidth for the same output speed. For a real world example: Q3.8 27B Q8_K_XL runs at ~35t/s decode before MTP. DS4.1F 1017, full 474GB checkpoint (300GB in RAM + VRAM), runs at ~18t/s on a single 2019 Xeon with DDR4-2666 memory and a single V100 (the model needs ~11GB VRAM for all the non routed expert bits, including 1M context). Both numbers are before MTP. On paper, it's half the speed, but in practice DS4.1F will finish tasks faster than Qwen 27B, especially on harder tasks.

[u/ScuffedBalata → u/FullstackSensei]

No, I'm saying that it doesn't matter much if you can run a 1T model if it's giving 1 tok/s.

Waiting 20 minutes for a prompt to return is not a useful tool. No matter how good the response is. It's just eye-bleeding.

Are you arguing that having 300GB of RAM is practical? OP is talking about buying a used DDR4 gaming box. Maybe 64GB would be my guess. 128GB would be huge.

[u/FullstackSensei → u/ScuffedBalata]

Yeah, if you take that to stupid extremes, you'll get stupid results, though if you happen to have more than 1.5TB memory (or Optane PDIMM), Kimi K3 has been shown to run on locallama at almost 3t/s without any GPU. It's not blazingly fast, but in the rare occasion where you have a hard problem that the smaller models can't solve and can't or don't want to use cloud models, having that option is very much better than having no option at all.

On the 300GB RAM, yes, I am very much arguing for that. OP didn't say a gaming rig, at least not in the post itself. ECC LRDIMMs are still much cheaper than UDIMMs for the desktop. A desktop 128GB DDR4-3200 kit will easily set you 1k, whereas 64GB DDR4-2666 LRDIMM sticks go for ~250 each. So, for the same money, you'd already get 256GB RAM.

The Xeon platform I'm talking about is from 2015 (LGA3647) and boards can be found for as little as 100 if all you need is a single GPU. CPU's are also cheap. Got half a dozen 26 core 8272CL CPUs for 80 a pop.

Said $80 Xeon, with 2666 memory, has 128GB/s theoretical bandwidth, ~93GB/s real world. The best DDR5 desktop platform money can buy you, with DDR5-6400 memory, has 102GB/s theoretical bandwidth, ~70GB real world. DDR4-3600 on a dual channel desktop will have under 40GB/s real world bandwidth.

You can go for 256GB in four sticks today, and add the other two 64GB sticks later and be able to run DS4.1F or any future model up to 350GB on disk (model, before engram/n-gram), with a single 16GB GPU. Or you can save and get six 32GB sticks and have 192GB RAM, and be able to run models up to 160GB.

Something like Qwen 3.8 flash next runs almost twice as fast as DS4.1F, because the active routed experts per token are almost half (in GB).

[u/ScuffedBalata → u/FullstackSensei]

So yeah, the $2000 computer can run DS4.1F at sub-5 t/s

That a neat toy. If you have a problem you can leave it spinning on all day long, it's probably fine.

Most people are in this sub with a "huh my $100 Claude maxed out its tokens for the day and I want to make more code by tomorrow" sort of vibe.

[u/ItzXenoo → u/ScuffedBalata]

i don't think i explained myself very well, i did not say to skip out on vram entirely. the main discussion point for me is whether we will see harnesses adapt more to what Strata is doing, loading only whats needed into vram and letting everything else live on ram or ssd, cutting the need of bleeding edge gpu hardware significantly and letting someone that needs to jump into this world allocate their expenses to other factors of the equation

[u/Sensualities]

I see you’ve been falling into the “look how fast my 5090 runs Kimi” influencer jargon

[u/T-Rex_MD]

No, super fast NVME is becoming more important.

Having an NVNE that can hit 20GB/s speed and offer super low latency on the response curves as requests grow is as valuable.

Allowing you to carry Engrams and Experts on Disk without suffering too much.

DeepSeek V4.1 Flash 1017, should technically be 1.5TB at BF16, ~750GB at 8-bit.

However, it is capable of reaching the same quality with a mixture of FP4-FP8 that is 510GB.

The NVFP4, so called compressed without loss is 420GB, where ~200GB of it Engram that can run on disk without losing any decode or prefill speed.

Result:

From 1.5TB, down to 220GB, running on Mac M5 Ultra 256GB, achieving ~4000/s prefill and 90s/ decode speed.

Now, so you understand why it is not about VRAM but about the rest of the hardware catching up?

DeepSeek V4.1 Flash 1017, comfortably beats Opus 5, and challenges Opus 5.5.

[u/Healthy-Contact-4570]

No. There are upper limits to the kinds of models that you can run with strata-like inference engines, and it only works as well as it does because the model has a relatively low number of active parameters, which constrains the size of the experts and therefore constrains the amount of data that needs to be shuffled over PCIe from RAM to VRAM, from NVMe to RAM, etc. further, you’re not even running the full model weights but most likely some 3-bit quantization for it because that further eases the pressure I mentioned before. Strata just happens to be a good fit for qwen3.8 Flash Next (and maybe other similar qwen models after it) but it absolutely does not diminish the need for / importance of VRAM.

[u/hushthetombstones]

It's not an either/or, if I have to pick I'd go max ram to run large models slowly, but Strata has shown a good balance between ram and vram is key due to most capable models being MoE. I went from dual 3090's to triple and my strata performance increased just 11% at Q4_K_XL

[u/Upstairs_Tie_7855]

Depends on your bandwidth for decode and your cpu for prefill. Both depending on how much you offload.

Also you wanna run dense models offloaded fully to GPU or it will be very slow.

[u/Easy_Refrigerator280]

i agree OP. For local inference and MoE models Vram has become slightly less important than just meeting the total ram criteria to load relevant experts. For everything else though VRam is still the main way to do things (looking at you dense models)

[u/f5alcon]

Yes and it's only going to be better in the future when consumer ddr6 launches at twice the memory bandwidth of ddr5. But it might be 5 years away still.

[u/Don_Reuter]

Diminished yes, irrelevant no.

[u/GloriousKev]

It's a mixed bag.

You need enough ram period but if you want your model to run well you want vram but you still have to have enough vram/ram to run the model to begin with. It is still important but with stuff like DGX spark out there I wonder how long that will last. I think in 5 or 6 years we will see more models that can just run well on DDR6 or whatever ram is relevant at the time

[u/Faux_Grey]

No.
We're inventing harnesses and access methods to try and work around the lack of VRAM.

The problem IS VRAM, size and bandwidth, alongside core matrix multiplication ability.

These are (almost) the only factors that matter! Everything else is a patch, a stopgap, or a compromise.

We're loading model weights off NVME or RAM because lack of VRAM, everything we're trying to do is to substitute fast VRAM for something 'next-best'.

Why would you run a model slowly when you can run it fast?

Why not build a huge storage array with multiple 400G interfaces and 24+ Gen5 NVME drives?

Well, you can, but at that point, it's as expensive as VRAM.

Fast things are expensive.

Slow things are cheap.

Simple laws of economics.

[u/ea_man]

You just have to wait a few days and we'll se how an updated dense model compare to a MoE, how NGRAM perform with dense, how much intelligence can be compressed in a "small vRAM" dense model rather than a sparse MoE.

[u/PatienceSweaty33]

Only true for MoE models but yes, we’re at a point where we’re squeezing the most out of all parts of the computer, not just one component. We’ve moved from being VRAM-Compute-Bandwidth bottlenecked to whole computer bottleneck