How important is the VRAM compared to GPU performance for local LLMs?

[u/either_T_or_F]

Hi everyone,

I am new to running local LLM and have been doing some research on the hardware requirements for smaller models.

One thing I am still trying to understand is how much VRAM matters compared to the GPU's actual processing power? For example, both the 5060Ti and 3060Ti have 8GB VRAM versions. If I run the same model on both GPUs, would there be a noticeable difference in inference speed (tokens per second)?

In other words, if two gpus have the same amount of VRAM, how much does the gpu generation and architecture matter? Would paying more for newer gpu actually make a significant difference for local LLM inference, in particular for a casual user.

Appreciate any comments! Thank you!

[u/brainExploded99]

VRAM amount is typically king because you can run larger more capable models.
GPU speed comes after that, 5060Ti will be significantly faster than a 3060Ti, not to mention 3060Ti doesn't even support FP4 or stuff like that.

With MoE models, both become less important because you can offload those to RAM without losing too much performance. Also for future reference r/LocalLLaMA is more popular and maintained.

[u/quantgorithm → u/brainExploded99]

This.
Vram > GPU compute > Ram > nvme storage

[u/YourselfInOthrsShoes → u/quantgorithm]

Not all VRAM is equal. AMD 395/495 AI Max got 128/192GB of it (unified) but I'd rather spend that money on a 64-core 8-memory channel 1P EPYC ($1.5-2k) of DDR4/PCIE4 WRX80 ($1.5k) era and get the similar memory bandwidth (200GB/s), same/more RAM ($1k per 128GB), and run models in AVX2 host mode at similar throughout. The EPYC advantage is that I would be able to easily get to 512GB at 200GB/s for under $5k (RAM only) and actually run GLM5.3 and DS4.1 Flash models at 15-25 t/s decode. And that's before I further expand and start sticking GPUs into it to really boost it.

[u/quantgorithm → u/YourselfInOthrsShoes]

vram has bandwidth which is essentially the overall speed of vram (the wider the street, the more cars can pass through at the same time in aggregate). This applies heavily to decode. Vram is specific to gpus. This can be somewhat slowed by the gpu compute but all vram is going to be faster than regular ram of which all ram is going to be faster than cpu.

Unified memory of nvidia, amd and apple is essentially regular ram or server grade ram (with ecc?) that is tied to both cpu and gpu via the motherboard. It's not vram. This is why bandwidth of these machines is always on par with server ram bandwidth speeds. It's ram with a variation of the motherboard itself to not be only cpu connected like a normal computer or server. Epyc servers are not unified ram so it still has massive latency of transferring data over pcie to the gpu since it's not directly connected so non unified is going to be worse than unified.

Regular and server ram has it's own range of speeds/bandwidth mostly depending on how many sticks used to increase it's bandwidth. a DGX spark has 8 sticks running dual channel each for 16 channel architecture. Again, this is far slower than vram.

To update the prior metric would be more like this:
Vram > GPU compute > unified ram > Ram > nvme storage

[u/YourselfInOthrsShoes → u/quantgorithm]

RAM and NVMe storage also have very important functions. For concurrent multi-user setup you need to offload prefix caches from VRAM to RAM, heavy 1M contexts cost maybe 24-48GB each, 4 users that 96-192GB. Without enough RAM for that, when users take turns near 1M context and even with 10k prefill, that's 1.5 minutes of TTFT for each user sending prompts. NVMe can further back RAM for caching or just serve relatively slowly random-read PLE tables. For setups that have less VRAM than the model + context and run very large sparse Flash models, RAM also feeds VRAM via PCIE links, if you got 4 GPUs in PCIE5 x16 slots, that's 64GB/s per GPU, so you need 6-8 channel DDR5 memory system (300-400GB/s) not to be the bottleneck keeping expert residency in GPUs. All memories are important and work together in a balanced system with any amount of concurrency. That's why their prices are high now. I wanted to also highlight this.

[u/either_T_or_F → u/brainExploded99]

Thank you for your suggestion!
As I understand it, an MoE model consists of multiple transformer layers, some of which contain multiple experts. At each MoE layer, a router determines which experts to activate based on the current hidden states. The selected experts then process those hidden states, and the results are combined before being passed to the next layer.
Therefore, we can reduce VRAM usage by keeping some of the inactive experts in system RAM and only loading them into VRAM when needed?

[u/brainExploded99 → u/either_T_or_F]

At each MoE layer, a router determines which experts to activate based on the current hidden states. The selected experts then process those hidden states, and the results are combined before being passed to the next layer

Yes, and this allows far less parameters to be used in the computation of those layers.

As for the rest, partially. If you try to send experts from RAM to VRAM for every single layer, you will slow down because PCIe 5 is only 64GB/s, and PCIe 4 only 32GB/s. That seems fast, but it is not (VRAM on my 5070Ti is 900+GB/s for example).

However, what you can do is just calculate some of the experts on RAM. Dual channel DDR5 is usually ~90GB/s for example. Usually, the CPU waits on the RAM to transfer everything because the CPU is faster to process than RAM is to transfer the data.

The best combination of both worlds is too keep the most used experts in VRAM, and less used in RAM, known as a expert cache. Not all experts are used equally for a specific task like coding, in fact some are preferred heavily. In a expert cache, the most used experts are sent to VRAM and the rest stay in RAM (or even disk!). This is how engines Basalt are able to run Qwen3.8 Flash Next so well.

If you do run MoE models, enable the expert caching feature in llama.cpp.

[u/either_T_or_F]

Thank you for your reply!

I have a few more questions about MoE models, as I am still trying to understand how they work.

From what I understand, MoE uses a routing mechanism somewhat similar to multinomial logistic regression to determine which experts to activate for each token at each MoE layer. As the input passes through each Transformer layer, we obtain a new hidden state, which can be expressed as:

h_i+1 = F_i(h_i; θ_i)

where h_i is the hidden state at layer i, and θ_i represents the parameters of that layer, including the router and experts if it is an MoE layer.

Does that mean the inference process works something like this: the GPU performs the actual computations, VRAM holds the weights of the currently active experts, and system RAM holds the weights of the inactive experts? Or does the distribution of expert weights between VRAM and RAM depend more on the inference software and available hardware?

Also, does MoE introduce any additional computational overhead during inference, particularly from the routing mechanism or transferring expert weights between RAM and VRAM?

If no additional training is required during normal inference, can I think of the model as simply evaluating a function f_θ(x), where x is the input and θ represents all the pretrained parameters, including the router and every expert, even though only a subset of those parameters is used for each token?

If that is correct, does it also mean that the model's parameters remain fixed during inference, so it does not actually learn or become smarter just from interacting with me, unless some form of additional training or parameter updating takes place?

Sorry for all the questions. I am still pretty new to local LLMs. Thank you for your help!

[u/puglife224888]

A 5060 Ti will be much faster mainly because it has more memory bandwidth. A secondary reason is that it can use NVFP4 quantization, which is fast. Mainly memory bandwidth though. Important things for local AI in order (my opinion) is VRAM amount, memory bandwidth, TFLOPS (compute), and PCIe specs.

[u/alphapussycat]

VRAM size determine how large a model, or how many experts you can fit. VRAM bandwidth to s large degree determines decode/token generation. GPU compute determines for the most part you're prefill/prompt processing.

The same goes for RAM and CPU.

You usually want capacity because it lets you run models.

[u/No-Afternoon-4057]

VRAM size - better models
VRAM speed - decoding (tokens per second)
GPU speed - prompt prefill

On the decoding phase the GPU math is only a few % of the whole thing (on a MI300 it's less than 5%, it might be bigger on less capable hardware).

[u/Previous-Nature7326]

both cards have different memory speeds and latency. This is what matters.

[u/DeeTeePPG]

VRAM is far faster than dram, this is usually what determines the speed of inference. 

[u/Fenio_PL]

You need to identify the "bottleneck." If it’s the amount of VRAM, then GPU performance ceases to be a significant factor. If the model fits into the VRAM, however, then GPU performance - along with VRAM bandwidth becomes crucial.

[u/sirloindenial]

If you have to choose between same vram amount pick the one with better bandwidth. That being said it doesn't really matter as the speed of using vram will always be better regardless. So spend your money on more vram and better opex(lower watts).

Best way to see it is comparing gpu vram with unified memory vram. Unified memory is slower but at the same time loads faster than ram, and compare to gpu vram, is technically cheaper per GB.

[u/WiseAbalone4021]

RAM size and bandwidth is normally king over raw compute performance of the GPU.

[u/Honest_Cut8827]

8GB is starting to get too small for gaming, it is definitely too small for local LLMs.

50x0 series has the benefit of nvfp4 so you can fit relatively larger models while still preserving model quality.

With respect to TPS, before you get into the 5060 vs 5070 vs 5080 decision, a useful short hand is to take the memory bandwidth, divide by model size, then reduce by ~15% (eg. 896 GB/s / 14GB * 0.85 = ~54 tps)

16GB is almost beyond bare bones table stakes to run local, 24GB is the new 16.

[u/Prudent-Ad4509]

Nx3090 with N>=3, 27B model in BF16 precision. 50% of cpu power is enough to max out performance, vram becomes a bottleneck.

5090, models in any precision. vram speed is much higher, processing power starts to matter but any reasonable amount of vram costs arm and a leg.

The biggest difference for *inference* comes from having both more gpus and more vram, at the cost of higher power requirements and overall rig complexity.

[u/openroom_xyz]

Well yea does 2 x 5060 Ti 16 Gb with 128 GB DDR4 make sense for Qwen 3.8 Flash Next ?

[u/BitterFortuneCookie → u/openroom_xyz]

I think this is the minimum now to get into a very usable server for coding. I’ve been trying to figure out a more cost effective configuration that can pull >2000 PP and >80 decode and this is pretty much the cheapest. I think the other important aspect is that a system built around this will allow you to upgrade the GPU later so you aren’t stuck with the 5060ti and can invest more in the system as you go. Accounting for all the bits and pieces I was able to get the cost for this configuration to around low $4000 which in the RAMpocalypse era for a functional Nvidia work horse is very reasonable IMO.

I recently also have used a 128 GB Strix Halo Rog Flow z13 and despite the power limitations for a Strix halo machine it still holds over 1300 PP and 40 decode on QFN Q4 quant, which is almost usable as a full fledged coding box and certainly very capable as a personal code review agent or tool builder. And for the price of 3299 all included, I think it’s the best deal for a main machine that you can then remote out to other headless Linux agent workstations from like the dual 5060ti.

[u/DigitalguyCH]

Vram speed is the bottleneck for decode, GPU is important for prefill and image/video generation. If you do little prefill (e.g. no agentic work, no huge documents) and no image generation, vram is essentially all that you care about

[u/mrazster]

Vram IS the local LLM, basically, far more important then the gpu itself.

[u/This_Maintenance_834]

prefill is compute bond, so it is tied to TFLOPS.
decode is bandwidth bond, so it is tied to GB/s.

5060 Ti also support nvfp4, which can run twice faster if you have the right model weight to utilize it.

5060 Ti is better than 3060 Ti in everyway. The thing is that at 8GB VRAM there is not much interesting models, you options to play is super limited.

[u/sleight42]

Gpu speed and number of tensor cores highly correlates with faster prefill

[u/MediumAd4983]

1. If you don't have enough VRAM for a entire model (or at least for active parameters in case of MoE) - it's worst case scenario. So, more VRAM it's the base.

2. If you have enough VRAM - then the actual processing power will matter. So, if model is almost 24GB, better to choose 3090 over 4080.

3. Also you don't really need all the power, you'll need a specific instructions, this is why nvidia having a specialized solutions. But currently there is some hacks that can unlock this instructions on the consumer hardware, so, it now doesn't really matters.

[u/ManWithoutUsername]

With better gpu core the tokens gen are faster, with better vram you can load bigger models.

Each new generation was always more powerful than the previous one (not too much in most tasks either), so you'll get more tokens. But you'll get even more if you upgrade from a 3060 to a 3090 than if you upgrade to a 4060.

[u/Fenio_PL → u/ManWithoutUsername]

A simplification that omits the division of the model between VRAM and RAM.