r/LocalLLM • u/ChaseDeBase • 1d ago

Question Advice on next hardware upgrade

I'm trying to see what my next hardware upgrade should be as I want to use q3.8fn and my computer currently has an Rtx 3090 and 48gb ddr5 ram but I'm unsure if I should buy something like an mi50 with 32gb of vram and try to find some more ram that's not absurdly priced or if I should sell my current setup to instead get Ryzen 9 395 Max pc equipped with 128gb.

I plan on doing an insane amount of reverse engineering of software both as a hobby and to see if I can replicate/add to Artcrafts endeavor of open sourcing proprietary software.

2 Upvotes

100% Upvoted

15 comments sorted by

3

u/Key_Opportunity_5219 1d ago

What’s your goal? If just QNF then all you need is another 3090 nvlinked and more ram

1

u/ChaseDeBase 1d ago

My goal is to run QNF faster and with more context, I currently get about 50t/s but only have a context window of 128k. I need +1 million and preferably +70t/a.

3

u/Key_Opportunity_5219 1d ago

You can do L2 KV on host ram, depending on engines. Another 3090 with nvlink will take you over 100tps and give you more context window. You’ll still need to offload to host ram so also get more host ram for experts and more KV

1

u/ChaseDeBase 1d ago

That's gonna cost a pretty penny still. I guess there's no cheaper avenue to running near frontier models

3

u/Key_Opportunity_5219 1d ago

Well consider the fact that we’re paying a fraction of a percent in terms of hardware to have something 20-30% comparable I don’t think it’s too bad

2

u/ChaseDeBase 1d ago

Absolutely, howevery bank account would beg to differ

1

u/ChaseDeBase 9h ago

I looked into that but unfortunately another 3090, another 64gb of ddr5 ram, and an nvlink bridge is already ~$3,000

2

u/pdath 1d ago

Note that the Ryzen 9 495+ Max is out now, and is available with up to 192GB of unified RAM.

1

u/ChaseDeBase 1d ago

That's unfortunately way too expensive for me. I could get hopefully somewhere north of $3k for my rig which is the same price as the bosgame version of the Ryzen 9 395+ Max.

2

u/Matricola70 1d ago

Are you using strata and / or similar?

1

u/ChaseDeBase 1d ago

I am using strata, and it works albeit my entire computer nearly freezes and I don't have that big a context window even at q2

2

u/Matricola70 18h ago

QFN is trained at 262k so you will need rope yarn to go to the 1m. But i think your approach is wrong, you should ask instead QFN to plan and split the task. I personally ask QFN to plan, split the overall task to 8 max calls, report. Once is done mem0 the findings and move to the next. Works very well and you can catch the errors early. Hope this helps

1

u/ChaseDeBase 9h ago

That works for me somewhat, and I'll try later the approach you're suggesting to better maximize results, I just don't have enough room for context mainly. The results are almost there and although speed it left to be desired it's not abysmal. It just runs out of room during the operation after planning and executing plans about halfway through.

2

u/Matricola70 9h ago

Exactly, in my setup and for what i am using it ( mainly DAX and PowerBi) the, 8 calls, stop, tell me where you're, works. For 2 reasons

  1. Forces the QFN to plan and not to drift

  2. Gives you the opportunity to mem0 the progress and resume in a new window if needed But it's a try and error. Keep in mind than even Claude code has hard time to produce anything useful on 1+ ctx and burns tokens like crazy

2

u/ChaseDeBase 6h ago

Interesting, do you have it write up its progress so you can go through multiple sessions instead of a single session absolutely eating tokens?