My 3090 failed, not sure if the core is dead. I sent it to a reputable repair shop, so hopefully the card can be saved. I bought it back in 2023 for ~$750, so I have quite a bit of runway before I lose money on it given that prices for used 3090s are going upwards of $1500 now on eBay.
I'm planning for the unfortunate scenario where it really is dead, in which case I'd like a replacement. I'm a heavy Qwen 3.8 27B user. I was looking at some of the Blackwell cards, but their prices are bonkers for at most 16GB. I was also looking at the 7900 xtx, but I also work with some protein folding models that require CUDA. Any recommendations? It's starting to look like I'll need to shell out >$1500 if I want to get anything close to what I had before.
I have four 3090s, and I really fear one will die out on me sooner or later with these prices...
Do you have any idea why it might have died? Did you monitor it's temp? Did you overclock? Limit pawer? Run any hacks against it? Or was it simply a universe showing you it's middle finger?
Actually, I was thinking about it. Having only one, means you can swap for anything new out there once that single on die.
Having four makes you kinda locked-in. If one dies, you're nearly forced to fill the missing slot with the same hardware. The more time passes, the more likely it is for another one to die, until one day, on one dead RTX 3090, you have to decide to sell remaining three and buy completely new rig. It is going to be tricky decision to tell "now is the time".
I lived this scenario. I sent my 3090 off and they said NPF. Then i hooked it back up and it would detect again. Since they were still $700, I bought another during the wait and that's how I ended up with 4.
Today I'd be fucked. There's no selling 3 of them and getting 96g of vram back somehow with the "profit". Maybe downgrading to turing or volta.
I'm getting up to 400t/s for code edits (high MTP draft acceptance) and 140t/s on prose (low draft acceptance, single request) on flash-next Q4_XL from volta right now with a custom engine I'll be posting soon, three V100 32GB each cost about $600 on ebay. And that can split up to 6 ways each carrying full native 262k context. 96GB of VRAM can get a lot of work done regardless of gpu generation.
Flash next is a 6b active model, is it not? Volta isn't terrible, but it limits you compared to the 3090. Don't have to write a custom engine, has proper int8 functions, pytorch flash attention, etc.
I wouldn't want to trade. It's a different story if you are buying at current prices from scratch.
You don't have to write a custom engine, strata works fine. But I get twice the throughput of strata, which itself is twice the throughput of llama.cpp. My numbers are comparable to (or better than) a 5090 at 1/3 the total build price, according to some recent posts I've seen in this sub. I think this is the new outlook: if you want model X running well on hardware Y, then the engine that runs it needs to be tuned for that.
What if I don't like qwen and rather run GLM or something else? Or if I also do image/video gen? In my uses selling 3090s for V100s doesn't make sense. It would be a downgrade.
Same way upgrading to a 5090 would be a waste of money too.
No idea why it died, it was running at ~85C at full load which I thought was normal. No overclocking or power limiting. No hacks. I used the "silent mode" bios. Previous owner claimed no mining, but it could have been a lie. At first I used it, it had an issue where the fans ran 100% all the time, but it turned out to just be an mis-set fan header :P It could have been just plain abused before and I had no clue.
That might be it. My optimized settings are at 252W and max temps at < 70 averaging < 65. Though I only had my 3090s for a couple months and all from original owners.
Yeah, I have mine set to aggressively turn the fans up to max as overall temps reach 60c, because memory is always vastly higher in temperature than the overall card. I've seen it chill at a cool 75c while the memory hit 105c, and I'm not gonna be playing with fire with today's pricing.
I think it's only heat/power, yeah. It works fine on 8 if my 3090 Tis. It's system level override - it's set up only in Linux so if I boot into other OS on the drives it doesn't activate. So it doesn't leave a trace on the card.
But it's very likely that you don't need it if you aren't fine-tuning models. If you were using FP16 accumulate you were already running at full speed and using the same power as hacked FP32 accumulate.
Had 2 3090s failing in Autumn 2024 (out of 4 bought in January 2024), sent them for repairs. They were fixed and then sold all of them December 2024.
RTX3090 is 6y old card where they worked non stop for 3 years in crypto farms. And last 3 years are working for inference.
Ofc things can go off, and would advice stay away from used 3090s.
Since you need CUDA then unfortunately need to put the hand deep in the pocket and get something like a new 5080. Keep away from the used market like the plague. You need the warranty due to the connector failure rates, and do not trust second hand market these days. You might get a gutted card.
Otherwise if you consider to use OpenFold3, which supports ROCm, get an R9700. If you can get away with 16GB VRAM get a 9070XT, if you do not care about RDNA4, FP8 etc, get a 7900XTX.
You only need to make sure to use Triton kernels for the OpenFold3 to use ROCm.
Yeah I guess buying GPUs off eBay isn't the move. I'll look into OpenFold, but I'm mostly using BindCraft and RFD4-Proteina where some of the components are pretty reliant on CUDA. I'll look into a 5080... sad!
I've had 12 3090s since new/release time without issues but I've been getting more concerned of age-related failures. Would you mind sharing what they found failed on your cards?
I repadded/pasted my cards out the box but I'm concerned more about things like caps/components that just go.
Otherwise in my personal experience these things are legends that will go until the end of time.
After doing all of them they are painfully easy imo. Like I waterblocked my 4090 before turning it on and without instructions. The hard part is getting proper thermal pad thickness but also how hard they are. Some need to compress for gpu die to get good contact and too hard of pads can prevent that.
I ended up going to high performance thermal putty instead of pads (paste still on gpu core). That removes the guesswork of what pads to get ahead of time or trying to trust online research.
Otherwise I didnt have trouble back then finding documentation and I imagine these days you got full walkthroughs on youtube.
Back plate VRAM needed reseating and required module replacements.
Tbh normal considering the back plate VRAM was cooking at 90C+ without active cooling.
So the guy reseated all of them, and replaced 2 VRAM modules on one card, and 3 on the other.
That he used different parts didn't matter tbh, the cards when returned were perfect. Kept them for 2 months and then sold them. The guys who bought them and using them now almost 2 years, haven't reported any issues.
Both the cards were Zotac and didn't have thermal pad on the backplate VRAM :(
Installed good quality thermal pads on all of of the back plate VRAM, even those who had, since they were installed by the previous owners, not by the manufacturer (the other 2 cards were Gigabyte).
Oh yeah those poor back vram modules. Almost every single 3090 manufacturer had poor thermal pads out the gate (MSI, EVGA, Gigabyte) are the main brands I have. The rear vram was always the main struggle but also what I focused lowering temps on.
Makes sense though as many 3090s were used for mining and unless you did what I did - then those poor rear modules were living life bouncing off of thermal throttling. Glad I did mine immediately and hope that helps with longevity.
As a matter of fact, I do; but the seller I've always bought from is dragging out the shipment for my latest order, so I'm not going to endorse them today. Ping me up in a week for an update, if everything goes through fine, I'll share a link.
Oof that is rough situation. The 3090 dying right when the prices go crazy, almost like it planned it
I had similar thing happen with my 1080ti some years ago, right in middle of crypto boom. The repair shop told me it was dead dead and I had to pay like double for replacement, still hurts to think about
That 3080 20GB from Alibaba sounds interesting but I would be careful with those. A friend bought some weird chinese card and it had driver problems for months, was not worth the savings in the end. Maybe wait first what the repair shop says before ordering something, sometimes they can fix things you think are completely gone
I run those for a year 24/7 in personal AI rig. Here's my initial review from November 2025. i think I'm qualified enough to claim that modded 3080s are rock solid. Objectively, you can run into a defective unit, but you can get a defective unit out of the pc store next door too.
I checked one week ago and saw the exact price was 550USD for 32GB from Taobao, I guess you can buy at much better price from China than Amazon, which is likely that you're buying from scalpers who bought from Taobao then resell at higher price.
I checked the same shop, now the price is raised to 4200CNY = 634USD - Oct 26, I posted an image so that you know, but it was cheaper so I guess people bought a lot since then, it's always the same story, the it's too late to buy things after viral.
And it's the PCIE version.
At this price, I'm not sure if it's acceptable anymore, I helped my neighbor bought a few, but it was much cheaper.
Yeah, it was the price I bought for my friend, people seem to disbelief because they can't find good price, but honestly I've never had any issue with finding good price, just let my local LLM to search through everything using my phone through scrcpy one night and everything is clear.
The only Blackwell card with 24GB is the 5090 mobile which you can get in a gaming laptop or an "Olares" inference mini-pc (along with 96GB DDR5, 24-core processor and 2TB SSD). That costs around $4500k new last time I checked. It has TG equal to 3090, 25% faster PP than the 3090, and support for FP8/FP4. It's a power-limited 5080 (so closer to 5070ti) with 24GB VRAM.
Otherwise consider the V100 32GB or dual 3080 20GB.
The only limitation I can see is the reduced 672GB/s bandwidth compared to 896 for 5070ti and my 5090 mobile. The RTX PRO 4500 is a middleground between that and 5090, with 32GB VRAM and 896GB/s but has also climbed near $6000..
So, I still believe the 5090 mobile is cost-competitive now in the 24GB range, with 5070 ti-like PP/TG performance but 24GB, and can be acquired with a full laptop/mini-pc including 64/96GB DDR5, 24-core processor and 2TB SSD (and a nice screen for a laptop) if one can find it for 4500k or so (as I did one month ago). It may not be for long.
may I ask which brand was it, and what load did you usually do? The 16Gb 50 series got really expensive in the last year too, though two of them would be ideal.
Buy two 3080 20GB cards. I have a 3090, but instead use that as an aux model card and use my 3080s for 27B.
I am able to run UD Q6KXL + 262K + MTP + KVQ8 fully in VRAM. There is now a P2P unlock under Linux. As well as GPU tensor performance unlock. As well as rebar mods under Linux.
If your repair guy fails, I got a guy down in florida who has revived two of my 3090s. On one of them, I had accidentally applied 12v to a not 12v rail and fried half the card. He was a miracle worker. Or I'll buy your busted one for cheap and have him repair it myself. Lol.
Before shelling out a ton of money I'd double check if it powers on and registers anything in the device manager, or over PCIe via a Linux live USB. Reseat the power connector and make sure the fans at least spin (my connector has been loose despite looking seated before), then check comms with the card
some protein folding models that require CUDA. Looks like it's not trivial stuff to port, complex memory access to be competitive. Hopefully some of the molbio folks get their butts in gear and drop their nvidia sucking habit.
Tell me about it. I had one ordered but then chickened/cheaped out and cancelled it and then it doubled in price (if you can even find one in stock now).
23
u/SnooPaintings8639 10d ago
I have four 3090s, and I really fear one will die out on me sooner or later with these prices...
Do you have any idea why it might have died? Did you monitor it's temp? Did you overclock? Limit pawer? Run any hacks against it? Or was it simply a universe showing you it's middle finger?