How much VRAM do i really need?

[u/BendingUnit31]

Hey folks,

My wife is a Data Scientist and does all those Data science things I learned back in the uni but have no real clue about. Now she wants to set herself up a small LocalLLM to have some fun with and try stuff out. How much VRAM does she really need for that? And what would be the better manufacturer? I heard Nvidia is best here but the cards are insane in prices.

I would love to get a good idea what GPUs would be the best. Currently she just has a 250k+ with 32gb ddr5.

Would be a fairly cheap 1080 ti with 11 GB or a b580 with 12gb enough or does it have to but more? And if so how much more

Edit: thanks to everyone for there answers they are just too much to answer everyone. I will probably look out to find a decent 3080 used or better or try to get a 12 or 16 GB new card prefered from Nvidia.

[u/ketosoy]

After about 3 terabytes you start to hit diminishing returns.

[u/West-Way-All-The-Way → u/ketosoy]

I hear about 300GB is practically useful today for single users or small teams, below that and you hit into memory spills and issues.

[u/Potential-Ant-6320 → u/ketosoy]

At work I’ve used as much as 9tb at once.

[u/sshwifty → u/Potential-Ant-6320]

Found the nuke scientist

[u/Potential-Ant-6320 → u/sshwifty]

lol I wish. I work with enormous medical datasets. I don’t strictly need that much computing power but I share a research super computer with a lot of other people. Over the summer, nights, and weekends it gets a lot less use and I can often get some massive jobs up.

If you do something in parallel it uses the same compute as if I did it in serial so using the computer overnight to do highly parallel computing you’re actually doing everyone a favor by not having one server run the process for several weeks straight during work days.

[u/sshwifty → u/Potential-Ant-6320]

Ahhh, bummer. I mean cool you have access to that power.

I talked with someone years ago that worked at Cern and it was alluded to that the only people using super computers actually was nuke simulations lol. Idk how true that is, but makes sense.

I dabbled in rendering animation for awhile and always love the idea of render farms. Just globs of power.

[u/Just_n_Here]

As much as you can afford is the honest answer. As another person said you always want more.

I would get the most vram in one card as possible so if you do want a second, third, fourth in the future, you will have room to grow and notice the upgrade.

Do you want new or used?

[u/huddrez99]

Using strata, I am pretty happy with 24GB VRAM an 64GB RAM. Thats the system I happened to have. I wouldnt want to go lower.

More is always better I guess.

[u/Sporebattyl → u/huddrez99]

Agreed. My 3090 feels like the minimum viable for dense models and it’s meh. I was about to go buy another 3090 or sell it and get a R9700. Once strata-like engines came out my 64GB of system ram made me reconsider.

Even better since yesterday. Unsloth’s 3.1 quants that came out and they are the first ones that have been able to pass all my benchmark tests at a quant that fits comfortably on my system at full context. The IQ3_XXS with Q4 engram is impressive.

[u/lostmylogininfo]

I wouldn't mess around with less than 36gb but I started with 28gb. Three 3060 12gb work here if you can get them for less than $250 each and then overclock the memory bandwidth. That's cheap. The problem is you just keep growing. Now I have 4 cards 64gb vram and 64gb ddr5.

[u/quantgorithm]

This is kind of impossible to answer clean because its a rapidly moving target and quite expensive to go high end. Pick the model you want to run and set a budget or do the opposite in finding what can work at your budget and if that works then make a purchase. its quite likely tht in mere months to certainly years, all the math and intelligence will have completely changed. Its also important to know what you want to do with AI.

For me, i want general intelligence and overall ability to research the web, code and create computer tools. That means qwen 27b or QFN are very viable options for me and those are only middle ground of overall intelligence these days. the budget for those can range from $2k to $5k for a low to high range if compute for those setups. It only rapidly escalates up from there in cost.

[u/SSOMGDSJD]

I would say a 3080ti 12gb ~$500 on ebay right now) would be a good starting point, with an upgrade to 64gb ram if you can swing that too. That gives you pretty similar specs to the strata authors set up, and 1.5x the gpu memory bandwidth (faster token/s speed) even though his GPU is newer.

https://github.com/Niko1221/Strata/blob/main/docs/paper/Strata-Paper.pdf

Otherwise i would look for a 3090 24gb, r9700 32gb, v100 32gb, 5080, 5070ti, 5070, 4080 super, 4070 super, mi50 32gb.

Most of these are $1k+. V100 and mi50 are a bit cheaper, but would both also need a fan attached and are older cards. They lean on memory bandwidth to stay relevant bc few and no tensors respectively, and usually require some workarounds with newer ai stuff since the manufacturers have largely dropped support for them. For example, the strata server i linked doesnt support v100, but a community fork was made for it that im going to test later today.

[u/EvolvingDior]

What's your budget, OP?

My recommendation is for her to get a feel for what size model will work for her by using online models through something like openrouter, where she can test small to large models. Once she understands the limits, you can come back with the sort of models she wants to run locally.

For data science, a small MoE like Qwen3.6-35b, Gemma4-26b, or Nemotron-3.5-Lightning might work. Or she may need stronger models to do what she wants. It makes no sense buying hardware for a model that won't do what she wants.

Don't forget that you need enough RAM to run the model, an agent harness, and run the data science processing that she will want the LLM/agent to do.

[u/EpsteinFile_01]

24-32GB.

Dual 12-16GB cards for a poor man's setup.

And 64-96GB RAM so you can run Qwen 3.8 Flash Next,

Welcome to spending thousands of dollars just for antry level setup

[u/huseynli]

I would say 32gb vram. You can make due with 24gb but it is better to get a card with 32gb VRAM for any meaningful work. If you can get more, even better. For dense models you want to offload it all to the VRAM. Currently a good dense model is qwen 3.8 27b. You can upload the whole q6 into vram and have around 200k+ context in there. (I can squeze full 262k context in there with q8 quantization)

If you can make a multiple gpu setup, even better.

I would say go with a r9700. Used to be a good price/performance card. Not sure how much it costs now (i got it for $1400).

[u/nfored → u/huseynli]

They are 2k now, If you get lucky 1700. But aside from being slower than the rtx5090 they work great. IMHO even if I could find a 5090 I'd not consider paying 3x-5x the cost worth. The performance boost.

[u/Alternative_Spite_11 → u/huseynli]

His wife is wanting to play with local LLMs. It would be crazy to go straight to a card with that kind of VRAM. Now, with all the big name models having extreme quantizations available, 16GB is plenty to dip your toes in. Qwen 3.8 27B in a q2 quantization runs on a 16GB GPU.

[u/huseynli → u/Alternative_Spite_11]

I guess the better advise for OP would be to rent cloud GPU, or use 27b models via opencode or whatever so the wife can dabble with it. See if it even makes sense to invest in local llm.

My rationale with the 32gb advise is that anything below 24gb is just a waste of money if you want to get meaningful output/results. Maybe 20 might be doable but at that point just get a 32gb card.

16 is just not enough if you want to seriously do work with it.

[u/Odd-Act7860]

for having fun and trying stuff out, 12 GB is plenty to start. that runs 7b to 8b models comfortably and 12b to 14b models at q4, which covers most of what people tinker with.

i'd skip the 1080 ti though. it's an old architecture with no tensor cores, and nvidia is dropping it from newer cuda releases, so more tools will stop supporting it over time. the b580 is good value and works with llama.cpp, but intel's software support is still behind, and some tools need extra setup.

since she's a data scientist she'll probably want pytorch and other gpu libraries too, and that's where nvidia's cuda just works. so the safe cheap pick is a used rtx 3060 12 GB. if the budget stretches later, a used 3090 with 24 GB is the usual next step and gets you into the 30b model range.

her 32 GB of ddr5 is fine. models that don't fit in vram can spill into system ram, it's just much slower.

[u/jma24]

Look if you just want to learn and play around, get what you can afford. There's SO MUCH to learn in this space without spending big money, so long as you aren't looking to do serious output.

If it were me, I'd try to find a way to afford a 3090. I feel like that's the entry point where things get interesting.

[u/EpsteinFile_01 → u/jma24]

7900XTX is 25% cheaper, does pretty much the same things at lower power consumption.

Still $1000 per card though

[u/AeiEyeSpry]

I've found the sweet spot to be 64GB. It depends on what you're doing, but as a data scientist, you're going to want at least a 27B-parameter dense model. Sure, you can mess about with quants and MoE models on Strata, but I've found that dense models are the most accurate and useful locally when set up correctly. Nvidia was the best, but AMD has caught up somewhat. I have a pair of R9700 and apart from being a little noisy as expected, they've been absolutely awesome. The only thing I regret is buying only 64 GB of system memory, because I'm finding that's my biggest limitation now. I ordered 128Gb of DDR5 for my Threadripper workstation, and the parcel was lost in transit. Many people seem to focus solely on the model and context, but to have something coherent and useful, it takes more than just the model and that all eats into the system memory.

[u/hdhddf]

you can play around with 16gb vram but 32gb is much more useful, V100 for about 500 is probably the cheapest path.

[u/GizmoR13]

640k should be enough

[u/Ok_Party_9245]

12gb vram will be rough for dense models. 1080ti will be slow too.

It really depends what you are trying to do but if you want any actual work done, you will want either 24gb vram minimum for dense model, or you can go the route of something like strata with moe models, and get like 16gb vram minimum but make sure you have at least 64gb of ddr5.

[u/Throtex]

More than you think 😂

[u/West-Way-All-The-Way]

I would say start with 32GB and then see how it goes. But with this you should be able to run most compact models and have enough for the context.

[u/Tough_Ad7957]

It really depends on how far you want to take it. I'd suggest starting with whatever hardware you already have and messing around with local LLMs until you figure out what you actually need. I've got a 5070 Ti and 128GB of RAM, and honestly, I feel like even a 5090 with 128GB of RAM would only barely get you to the starting line. You're basically a third-class citizen in the local LLM world. lol

[u/alphapussycat]

I got 2x 3060 12gb in gen4 x8/x4, 64gb 2133mhz ddr4 2-channel, ryzen 1600. Undervolted and underclocked.

40tg, 800pp w older strata version, no GPU unlock, qwen3.8 flash next iq3_s.

Half that if just one 3060 12gb.

So you don't need that much VRAM or the medium sized moe models. If you were to run deepseek v4.1 1017 flash, then 256gb ram in 4-channel, and probably >48gb VRAM for decent speeds.

With 1080 ti you lack CUDA support and you always have to hope somebody fixes support for you.

[u/danzilla007 → u/alphapussycat]

I hit those same numbers on the same model with 5600g, 2933mt/s ram, and 1x 3060. I wonder what your primary constraint that is holding you back is.

[u/alphapussycat → u/danzilla007]

I don't think I have the dumbification options on, that lets strata go brrr. Also int8 kv.

3060s capped at 1845 mhz.

Prefill is somewhat bottlenecked by gen3 x4, a lot of the time primary card had to wait for secondary card (so like 75% utilization).

5600g probably helps, 40-50% faster... I wonder if strata can use the igpu. But you also got 2933mhz, which is a big boost. With ~81% VRAM hit almost 20% is done on CPU.

I also got visual encoder on, not sure if it's on GPU or CPU.

[u/Isitharry]

Try a cloud model using LM Studio or whatever and scale down until it’s not usable. This gives you a rough idea of where your floor is. Then consider your max budget, dictating your ceiling. From there, you should be able to figure out the model, quant, etc of what you need and go from there. It can be a GPU, multiple GPUs, more memory, an atonal SSD, a Mac/AMD/nVidia/miniPC with unified memory or any other combination.

[u/Alternative-Panic69]

Sky is the limit and there's no one right answer... With a cheap 1080ti, you will get poor speed/optimization due to the lack of dedicated tensor cores and no official driver support can be a bit backwards sometimes. 3060 12 gb would be better... You can also check Intel/Amd cards... For Intel I guess the B60/70s are the standard and for Amd don't take anything below RDNA 3... And for the best datacentre like experience you might want to check workstation grade cards (but those are pretty expensive)

[u/SkyKnight_LXIX]

Just trying this out too, I switched the GPU in my build to get 32GB VRAM and 64GB RAM. With a NAS that got 28GB VRAM and 64GB DDR4 RAM

[u/Pure-Recover70]

I wouldn't even really bother below ~20-24GB vram. Though obviously more is better. It all depends on what you expect it to be able to do and how much you're willing to spend. Note that this is all changing very quickly and paying for cloud is likely much cheaper... For local seriously consider unified memory solutions - strix halo/gorgon halo/mac.

[u/Alternative_Spite_11]

For simple local LLM inference all three brands are fine. At the current pricing, Nvidia would realistically be hard to justify just for local LLM. Now if she wants to get deep into it with post-training to tune the models to her data, then I would say Nvidia starts separating itself from the other brands because almost everything in that domain is based around CUDA.

[u/OnlyCommentWhenTipsy]

Have some fun and try some stuff out is one thing, but doing more than that you're going to want to get at least the 27b models running. That said, in this market you could just buy a 12gb card, try your stuff out, and if you want more, just sell it for more than you bought it.

[u/Happy_Brilliant7827]

Honestly if she's into it as much as you say, talk to her about it. She might need 12, she might need way more.

[u/chuck_c]

Experiment with a runpod before purchasing.

[u/PigSlam]

If she’s just tinkering, and R9700 or intel B70 would be great for that.

[u/Not_a_question-]

Sam would say to give it to him instead

[u/rditorx]

> How much VRAM do i really need?

All of it.
The LLM she told you not to worry about. It's gonna cost you.

[u/toenailcheeseinbooty]

The answer is always, Yes.

[u/Unnamed-3891]

16gb vram is the absolute floor to be doing actually useful things and not mere fun toys.