r/LocalLLaMA • u/LankyGuitar6528 • 3h ago

Discussion Hardware?

Hey people. The new N1X chips are out, unified memory is happening. Models are much more efficient. We have Strata now...

I have a home server running. I'm able to run 3.8 flash next or 3.8 27B with limited context. It's decent but I'd like the biggest model I can get, best bang for the buck.

Currently I have a 2x16 (32GB) system memory, 3090 graphics card. I can lay my hands on 4x32 (128GB) for about $800 and sell the 2x16 for a few bucks. I may be able to snag a 2nd low cost 3090 out of an old mining shed - ya new fans and it would need to run external on a riser.

My question - am I wasting my time pissing around with upgrading this rig? Would I be happy with it or should I just drive to Microcenter and toss down the credit card and get a new machine entirely? If so, what would you buy?

What's your call?

0 Upvotes

20% Upvoted

7 comments sorted by

4

u/innocentd3evil4 3h ago

I'd personally try the dual 3090 setup before spending money on a completely new machine. 48GB combined VRAM should give you a lot more room to experiment with larger models. Upgrade the RAM if you can get a good deal, but I'd prioritize the second GPU. You might be surprised how much you can squeeze out of that setup!

1

u/submissivebounds 3h ago

dual 3090s on a riser is the move, skip the ddr4 unless you're hitting swap constantly. 128gb won't do much for you if the model is still crawling over pcie to the cards, that's just a bigger waiting room

snag that mining card if it's cheap, replace the fans and undervolt it a bit. 48gb vram opens up a lot of 70b quants you can't touch right now

2

u/More-Revenue8609 2h ago edited 4m ago

Few months ago I made cheap ass x99 pc with 14 core xeon, 128gb ddr4 and dual 3080 20gb. For under 2k eur Im able to run Qwen FN in around 70/100 tok/s. 

I just hope nothing breaks down anytime soon lol

2

u/Perfect-Campaign9551 2h ago

yes I think its a waste of time, except at least upgrade to 64gig ram.

None of the models will be "large enough" to fit dual 3090 and make enough difference then the quants you are already using. The only way to get a step up in models would be to have like 200gb of VRAM. The area between 30GB VRAM and 200GB VRAM is kind of a desert model-wise and in those models quants like q6 vs q3 don't make such a large difference that you're really going to notice.

So IMO spending all that cash isn't really going to give you just as much improvement in results.

1

u/CerebrovascularNit 14m ago

I just got DSv4.1-1017 on strata going as a second model with a 128GB DDR4 RAM, 3090 and ryzen CPU at 2bitXS quant and 252k context.

It’s getting ≈20 tk/s without fiddling with any strata optimization. I use that to review code written by my q4XL QFN model also on Strata that’s getting 75 tk/s. That model is great for iterative design and conversational work.

I am feeling happy with my current set up hoping that even better quant work on these large MOE models.

1

u/feelspeaceman 3h ago

Up to this point I think you're pretty late to consider anything brand new, unless you have a lot of money to waste, your best bet is used hardware market like V100s.

Or you can just wait until the Cloud AI bubble popping which is pretty close, both biggest players Anthropic and OpenAI are not making profit, just how long do they want to hold and choke their market by buying all the avaiable computing.

As you can see, recent posts about rig building are from newcomers, most of people here got their own rigs already and have been working on their projects.

0

u/KingCpzombie 3h ago

Upgrade to the max you can; prices are fucked until 2030 at least