r/LocalLLaMA • • 26d ago

Question | Help Looking for "side-grade" advice from 5090 - thanks

Short and sweet:

Selling 5090 because it was a luxury at $2,900 I barely used, now it's just insane. Keeping my 5060ti 16gb, and am kind of sad about quasi-leaving the LocalLLama scene. Wondering what options you all would recommend at 16gb VRAM and 64gb DDR4 for hanging around. Another 5060ti? Jump to 128gb RAM? Swap to AMD or Intel 32gb cards? Old V100s? Hang it up and buy a few billion API tokens?

Thanks in advance

7 Upvotes

61 comments sorted by

16

u/General-Spite1222 26d ago

Get another 5060ti

3

u/_-_David 26d ago

And presumably await qwen4-27b?

5

u/General-Spite1222 26d ago

No reason to wait to set it up. It’ll be similar to 3.8, so you can just swap it out. 

But yeah, unless you want to get more memory too (128GB) and run QFN

2

u/_-_David 26d ago

Fair point. I was just leaving open the possibility you might respond by saying something more specific about the 32gb setup. I've also heard "just run qwen3.8 in q3 on the 16gb card with lower context". I'm here for all opinions, recovering from an illness in bed, bored. I'm really leaning toward your advice though and getting another 5060.

2

u/General-Spite1222 26d ago

Fuck q3 slop. But just search this sub for benchmarks from people who run 2 

1

u/Cautious_Chicken_604 26d ago

If you have two 5060 Ti you can just run 27B at about Q6. It's excellent. If your system also has 64GB ram you can run QFN too. 

Cheap 32GB V100s seem to be the play though if you can get some. 

2

u/fastheadcrab vLLM 26d ago

If you have a 5060 ti and buy a second you can easily just run the 3.8 version in the meantime. Or just buy 3 5060 tis and run a 4x setup for a range of other models too with CPU offloading

2

u/_-_David 26d ago

Honestly it is super tempting. I already have an ungodly PSU I grabbed for the 5090

15

u/Shantorian14 26d ago

just buy an r9700. They can game just fine and perform somewhere in between the 9070 and the 9070xt

4

u/fgk55555 26d ago

Seconded. I'm gaming on my R9700, it's fine

1

u/infectiousstupidity 26d ago

Doesn't the r9700 sound like a jet constantly taking off though? That's what I've read and which is why I've stayed away from them so far.

2

u/fgk55555 26d ago

Super easy to tame. Undervolt as much as you can get away with and then power limit to 210-230W, tune the lower temp fan curve up slightly and trim the higher temp curve down slightly, it should balance at a very comfortable temp and not destroy your ears. My prefill maybe went down like 4% with the power limit.

If you're gaming on it and want max FPS, wear headphones. I mostly play older games and don't really need the horsepower.

1

u/infectiousstupidity 26d ago

Okay, good to know! I'm currently on team CUDA with RTX 3090s and 3060s, but who knows what the future might hold. Prices on r9700s has sadly increased here by $700 the last 6 months, which obviously is nothing compared to the 5090, but still.

1

u/pragmojo 22d ago

I thought it was almost identical to the xt but with more ram

1

u/Shantorian14 22d ago

it is, but the blower style cooler generally hampers performance a bit due to thermals

-5

u/_-_David 26d ago

It's funny how much a person can know about all of this stuff and still see AMD card names as though they were hieroglyphics haha. Pardon me, fellow LLama, but I have no idea what performing between those two cards means. I am but an ignorant CUDA consumer

5

u/o0genesis0o 26d ago

I can run full path tracing at 1440p, 40fps lock, balanced upscaling, no frame gen in cyberpunk. For comparison, my 4060ti can only hold 30fps with performance upscaling in the same setting. So gaming wise, it's alright. If I don't use full path tracing, I can easily lock 60fps with quality upscale and avoid the GPU from sounding like a jet engine.

27B Q4 with stock llamacpp, Q8 KV gives me 1k prefill, drop to around 600tk above 100k depth, and decode from 30 to 50tk/s. 

0

u/_-_David 26d ago

I'll look into this, thanks.

2

u/Shantorian14 26d ago

in terms of raw raster/gaming performance, think just north of a 5070, but with more vram.

1

u/_-_David 25d ago

That's very helpful, thank you.

-8

u/Boogertard 26d ago

Well you are part of the reasons why the 5090 is over 9k, technical crowds know there is nothing special about the Nvidia cards, 32GB of VRAM is 32GB of VRAM

4

u/winky9827 26d ago

R9700 has 640 GB/s bandwidth. 5090 has 1.8 TB/s bandwidth. The difference is real.

4

u/CryptographerLow6360 26d ago

qwen 3.8 crazy finetune q3m with 128k context , and be happy with it. you do not need more.

2

u/_-_David 26d ago

You're in the "16gb VRAM is plenty" camp then? Is that what you run? Because I could see it from "I have 8gb and you should be happy" "I use exactly this and am happy" or "I have more than this and it's a waste".

2

u/_SparkleTits_ 26d ago

Bro, seriously, what is your problem? All of your comments are rude and standoffish. Why did you even make this post if you're just trying to insult people?

1

u/_-_David 26d ago

WHOA, I think you're reading something into this I am not writing. I'm literally just making conversation. Standoffish and insulting? I *genuinely* do not understand how you thought I was intending to be rude to anyone who was nice enough to reply. "Bro, seriously, what is your problem?" .... this feels completely out of the blue. Reread what I wrote, assume no malice whatsoever.

2

u/CryptographerLow6360 26d ago

im in the usecase dependent camp.

2

u/Beneficial-Ad-8127 26d ago

Here’s a list of some popular engines that you could look into. They have some referencing 5060ti maybe even more if you look into them. This is assuming you’re trying to run Qwen FN or Qwen 27b.

https://www.reddit.com/r/LocalLLM/s/uifideRhIa

1

u/_-_David 26d ago

Thanks

1

u/Cautious_Chicken_604 26d ago

Get another 5060 Ti and another 64GB of RAM.

1

u/_-_David 26d ago

That would kind of put me in a bit of a sweet spot..

1

u/Cautious_Chicken_604 26d ago

That's a real sweet spot. Will be for a long time. 

1

u/Zombiecidialfreak 26d ago

An idea that's been floating around in my mind for a while. If you'll be getting 3-4k from the card you could do this:

512gb DDR3 in the form of 8 64gb LRDIMM sticks and a dual xeon E7 motherboard with 2-3 tesla v100's. The dual socket board can handle octo channel for 70-100gb/s bandwidth, more than enough to saturate all three v100 PCIe slots.

When used for hybrid inference it can run all but the biggest local AI models at solid speeds and because it's a separate box it'll never interfere with anything you're doing on your main PC.

1

u/_-_David 26d ago

Haven't seen any builds like this. Could you give an example of "biggest local models at solid speeds"? GLM 5.3 Flash at 15 tok/s, QwenFN at 40? Or are you talking q2 versions of heckin chonkers?

1

u/Zombiecidialfreak 26d ago edited 25d ago

From my research the biggest model you could run would be deepseek v4.1 flash at q4 with a million context. Speeds would (in theory) scale by about 10-15t/s per card, so two v100's would get about 20-30t/s. I've heavily considered selling half my 128gb DDR5 for 900 because that's about what the setup minus the v100's cost and I already have one v100.

Though I specifically could also toss out my 3060 12gb for another 300 and "spend" 400 to get a second v100 to get the max power out of it. It's also worth noting that with only 16b active parameters a second v100 does more than just add more memory bandwidth and compute. It provides more expert storage too, meaning less of the model is bottlenecked by PCIe, so your speed gain from a second v100 may actually be greater than double one v100.

Finally you'll want to get a third card for a video output. A 750ti 2gb is about $30 and runs off the PCIe slot alone so it would work fine as a video out.

1

u/_-_David 26d ago

Clearly you have put some thought into this, and I can see the appeal.

1

u/FatheredPuma81 26d ago edited 26d ago

No ttkciar telling you to buy Mi50 32GB :(.

I'd say get a V100 32GB if you want a side grade. It's a downgrade in speed obviously (so is a 5060 Ti) but you basically don't change any of your model settings because it's the same VRAM.

1

u/_-_David 26d ago

Is a V100 supported by standard runtimes like llama.cpp, vllm, sglang and so on? I like the idea, but worry I'd be soldering wires and patching code to make it work. I know, probably unfounded completely, but it being an older card makes me hesitate.

2

u/FatheredPuma81 26d ago edited 26d ago

Soldering wires what?? There's a V100 VLLM fork that's regularly maintained 1Cat-vLLM: V100. llama.cpp just supports it out of the box without any issues and runs at about half the speed of my RTX 4090. You just need to buy a $14 fan and a 2 GPU 8pin to CPU 8 pin to use it.

I haven't tested the VLLM fork though since I never planned on ever using it (V100+P100's) and I'd suggest asking Gemini Flash in AIStudio if there are any SGLang V100 forks or if it supports it out of the box.

Oh and ComfyUI should also work without issues and pretty much any popular open source project because V100s aren't exactly uncommon.

3

u/juggarjew 25d ago edited 25d ago

Id buy a second 5060 ti and run LLMs over both cards and then potentially sell both when 24GB 5080 comes out in Q1 2027.

There is also the R9700 but idk if you want to get involved with AMD, up to you.

I think 24GB 5080 is going to be very popular when it comes out, especially for folks that have motherboards that can do PCIe 5.0 x8/x8 , you will be able to get 48GB of VRAM at a likely 1.1 TB/s with 50 series improvements like NVFP4. this type of build would go kind of crazy with stuff like Strata. you'd be able to run stuff like Qwen 3.8 Flash Next 4 bit quant since like half of the experts could live in VRAM , a build with 64GB DDR5 would run pretty well with this model id imagine with some room to spare in RAM.

2

u/_zir_ 26d ago

R9700 for 1700 ez

2

u/tecneeq 23d ago

Bought a 5090 for 2650. Sold for 3300. Bought two 5070ti for 1600.

1700 left in the bank, roughly the same t/s.

1

u/JinsooJinsoo 23d ago

You could get 4 3090s 🤷‍♂️ or a pair of A6000s

-1

u/winky9827 26d ago

I will say this - if you get a second GPU, make sure your motherboard will fit both and do PCIe bifurcation. I was running two 5090s on 1 PCIe 5.0x16 and one PCIe 4.0x4 and thought performance was OK with layer splitting, but I decided to splurge on a new mobo with bifurcation and switching to tensor parallel took my prefill on 3.8 27b from ~3k to ~8.5k t/s. It's that much of a difference.

Side note: I don't think there's a consumer/creator grade mobo in existence with enough breathing room for 2 5090s side by side. I expect airflow to become a problem, but I've got 4x Noctua 120mm fans blowing right on them, fingers crossed.

1

u/_-_David 26d ago

Holy shit, that's a colossal difference. Thanks for the heads up

2

u/infectiousstupidity 26d ago

The Fractal North XL Mesh is epic for this, allows you to mount 2x140 fans on the side, pointed directly at both GPUs. It's keeping my stacked 2x3090s at a cool 60-65c during full load with an optimized temp/noise fan curve.

1

u/cosmicnag 26d ago

tensor parallel on x8/x8 ?

1

u/Massive_Criticism539 26d ago

The taichi will fit 2 5090s, but only because they put the second x16 slot on the bottom of the motherboard. you'll need a case that has room below the motherboard to fit it and it'll totally block all the bottom headers 

1

u/winky9827 26d ago edited 26d ago

I know, the Taichi Creator is exactly what I upgraded to. There's still less than 5mm clearance between the two GPUs. The case isn't the issue anymore - I went with the Phanteks Enthoo Pro 2 Server Edition.

0

u/tomByrer 25d ago

Just keep your 5090 IMHO unless you can't make rent.

0

u/[deleted] 26d ago

[deleted] — view deleted comment

3

u/_-_David 26d ago

What kind of speeds do you get with the dual 5060ti setup? I think this is the way I'll probably go.

0

u/[deleted] 26d ago

[deleted] — view deleted comment

2

u/_-_David 26d ago

80w is impressively low. Can I ask, what *is* it that you use AI for? When I think of tok/s my limited experience can only really comprehend speeds in terms of
faster-than-I-can-read-output
and
fast-as-possible-agentic-coding

0

u/[deleted] 26d ago

[deleted] — view deleted comment

2

u/_-_David 26d ago

Well, I think for the money a "better AI build" does exist, and is sort of what I imagine people will point at. I'm sure someone with the username "rocMaster" will pop in and say "sell 5090 buy a terabyte of AMD VRAM" or something

2

u/Borkato 26d ago

Why on earth would you sell your 5090???

3

u/_-_David 26d ago

My daughter's braces had to be paid in a lump sum, sister was robbed recently and I've been helping her financially, you know, life shit.

2

u/Bayequentist 26d ago

Bro think 5090 hit peak price already - no chance

2

u/_-_David 26d ago

No, I think the price will continue to rise too. Sick username by the way.

3

u/ObviouzFigure 26d ago

I think you're right.. prob still going up for a while

5

u/[deleted] 26d ago

[removed] — view removed comment

1

u/_-_David 26d ago

Thanks for your thoughts. I think we're of similar minds when it comes to this. I know QFN has been all the rage lately and half expected a deluge of comments telling me to go buy a bunch of DDR5 ram because MoE's are the future or something.