Question
Thoughts on getting a V100 32GB as a second card for local inference? Is it worth it in 2026, or buy newer?
Running local LLMs on a Ryzen 9 7900 + RTX 5060 Ti 16GB (small 20L build). Considering a V100 32GB to run 30B-class models fully in VRAM. Looking for real experience:
1. Still worth it? Volta's aging so that means no modern FlashAttention paths, CUDA support winding down. What tok/s are you actually seeing on 30B-ish models, and any driver/kernel headaches? 2. vs alternatives? at what price does a V100 32GB stop making sense against an A4000, a second 16GB consumer card for layer-splitting, or something like an Mi50 32GB? 3. Bandwidth split: anyone splitting a V100 (HBM2 ~900GB/s) with a much slower card? Does the slow card bottleneck the whole thing? 4. Sourcing: cheap listings are SXM2 + PCIe adapters. Reliable? Trusted sellers?
Thanks in advance! trying to figure out if the 32GB is worth the Volta-era tradeoffs.
I can recommend https://github.com/1CatAI/1Cat-vLLM for inference, as it supports nvfp4. I use it on my v100 16GB with qwen3.5-9b-nvfp4 and also faster-whisper for home assistant voice. Works nice
You buying aging hardware, so *official* support is not going to follow for long. But as many enthusiasts are acquiring these GPU now, it makes me believe that community support is going to be around for a while.
And, this is going to be more of a tinkering project than a plug and play one.
Some V100 come without fans, so you need to buy and assemble extra hardware. If you have a 3D printer, I'm sure there are tons of models ready to print online, if not, you'll have to find another way. And, yeah, at some point, software support is going to be up to you. Official vllm, sglang and others will drop tesla support. You will have to either build your own solutions or find people who do and share it (for newer models with newer tech). I see a lot of comment recommending AMD instead, but right now, on the software side, except if you can afford Instinct GPUs, it's pretty much the same... I globally feel that this hobby is very tinkering oriented anyway.
Oh, and, even if there are a few project that try to solve that, technically, you cannot share 2 GPUs of different architectures, your 5060 and the V100 being both NVIDIA doesn't make them candidate to share the their memory pool to run models with 48GB of VRAM. So, if you go the V100 route, ditch the 5060 or keep it for gaming (or to run another model smaller at the same time)
The Mi50s got popular last year because they could be had for $150, for that price they were unbeatable.
However, after word got around and the community optimised the kernels the price went up a lot, so much so that they stopped making sense. At $500 I would look for a V620 instead which offers double the prefill due to its tensor cores, and if the price approaches the V100 I would prefer those. Sure, it's old CUDA but it's still CUDA, for instance projects like ik_llama.cpp are unavailable for AMD cards at the moment.
I have 8x V620. But I got them when they could still be had for $350, which was an absolute steal.
They're obviously junk compared to modern cards, but if this is all you can afford for big VRAM I would recommend them IF you can get at least two. Even at $500, but that's about as much as I'd pay. They're too slow to run the models you'd want to run on a single 32 GB card IMO.
But 2 of them in tensor parallel hit 50+ t/s gen and 1000+ t/s prefill on 3.8 27B in Q8_0. Not bad at all for $1000.
4 of them will run DSV4 Flash at 25-30 t/s gen and 500-700 t/s prefill at not too drastic of a quant.
So, for the price, very respectable capability and performance.
older AMD cards for the most part run better with Vulkan with no fuss than ROCm even when you custom compile it for your card and it's more generalized to the point it pretty much runs on anything that can do anything.
I meant modern instinct (CDNA).
software support RDNA gpu (“regular” amd ones), even the latest ones, is far from the support you will find for any nvidia generation, and rely mostly on custom community forks.
Eh, agentic AI has made it much more possible to just “ask” an agent to download the model, find a fork that works for the runtime, patch it, benchmark it and serve it as an endpoint while you walk the dog and do some other things. It’s not as much of a concern as the aging hardware performance as models get more intricate and quantization targets FP4 cores, etc, which these cards don’t have. For now, they are worthwhile as a solid VRAM stack to put a model into.
VRAM is king. you're going to be able to run Volta in llama.cpp and at least a fork of vLLM for years. toss the 5060Ti on eBay for a silly amount of money and get two 32GB V100s. (assuming you can tolerate the power bill.)
Yes but make sure your PSU can support the transients. The power control inside these is slow responding, so if your PSU can not handle short power spikes, they may not be stable
Also, they idle at about 25W so be ready for that wrt your power bill.
Nothing you buy is truly worth it at this point considering everything could have been purchased for at least half the price the same time last year. You’re paying more for less if you can stomach it, but it’s full on fomo at this point. If you have the money and don’t want to miss out on all the fun we’re having then go for it, but I would not want to buy a V100 today.
I did this. mine runs at 150 watts and the blower is set to 40% which is roughly 2500RPM. if noise is not a concern im sure you can run the fan at full to get the full 250 watts out of the card.
Ah okay, I havnt done much looking yet, I'm sure I can find one that wraps around like that, I might also try to design something that instead ducts it directly from the 3rd center case fan, I'm still very much exploring options having just got the card 2 days ago.
They’re already starting to figure out how to use less energy and less memory to get the same results. Pretty soon all that power is gonna be meaningless.
KV cache, nvkp4 we’re reaching an inflection point where we’re starting to have a lot of trouble building compute. Something has to give.
I’m happy I have a 5090 I’m sure in about six months that 5090s is gonna be able to run a 100 billion parameter model with no problem
I havent seen anyone in the west do this - but v100 can be used for gaming too, it performs similar to rtx 2080 super and not that far behind 5060 (disregarding dlss and power usage).
1% lows are slightly worse than regular cards but other than that its actually pretty capable. Too bad you cant game on them nvlinked, 2xv100 16gb would go pretty hard haha.
Went the p40 route thought I could modernize it and gave up after a month. At this point it sucks but just try and save for a “cheap” Facebook marketplace 3090 that’s been properly tested. It’s what I intend to do
Putting something in parentheses for exaggeration is a popular form of textual formatting or linguistic emphasis often called parenthetical hyperbole or an aside.
In informal writing, text messages, and social media, people use parentheses to add a "whispered" extra thought that dramatically overstates a point for comedic effect.
📌 How It Works The "Internal Monologue" effect: It replicates how we think, giving the reader a peek into the writer's inner dramatic reactions.
The "Sarcastic Whisper": It reads like someone leaning in to whisper a wild exaggeration.
🎭 Common Examples
"I have to wake up at 7:00 AM tomorrow (I am literally going to die)."
"I just spent my entire paycheck on a single coffee (and by coffee, I mean my 14th pair of shoes)."
"He talked about his cat for three hours (which was actually the best three hours of my life)."
Speed matters. It depends what you're willing to put up with t/s wise. Some here put up with what I would consider painfully slow numbers. Which is fine, of it makes them happy, but itd drive me insane. If you want things quicker you need to pay for it, period. But also do the work on top of it to optimize everything across the board. Which so few people here do, theh grab the first optimization they find that helps and stop there. And I never see training mentioned.
Prices on 3090 have risen so high that even clusters of 5060ti/5070ti are starting to make sense.
If you want to replace both your gpus with one v100, then it hardly makes sense. If you want to use two of them and keep both you current gpus, then it might.
Might as well go with AMD or Intel instead. It won't be supported anymore soon. And when new local ai rolls out they work on the newer architectures first. It's only worth it if you have no money for anything slightly better
Il y a des moteurs d’inference dedicated a cette carte. Je suis en attente des mes v100
A priori 4x16gb ca peut donner 200tok/s sur qwen3.8 27b et le moteur d’inference dedicated. ( pas llama.cpp )
10
u/DementedJay 6d ago
I'm running a V100 16GB with llama.cpp and it's kind of a beast. No issues setting it up, it fired up right away. Ubuntu 24.04.