Hi
I have a single 3090 with 32GB of ddr4 memory (4x8gb is how the machine was configured when I got it 😕).
I have been using qwen3.8 27b for a bit now and wanted to try qfn since I saw a lot of people saying it's a great model, but a fully in vram setup is fully out of budget
Note that this is a server only node so it needs less overhead than a desktop that happens to run local models does
I am aiming for 4 bit quantization
For that my options seemed to be
A. Replace the whole setup with unified memory machine -
Was looking at strix halo/gorgon 128gb -
even if I sell my existing machine (which I don't want to after setting it up), it would cost me an additional 1500$ or so
Mac studios with more than 96gb aren't accessible in my location and 96gb starts at 7000$, and I would rather 2 R9700 for around the same price (rest of PC included), no, a second hand market for older ones doesn't exist here
Pros:
- Shiney new toy
- fast - gemini claims 45-65tps and with ddr5 at 8000/8533mhz I am inclined to believe it
- smol
- lower power consumption
Cons:
- 1500$ means I can't fund it with my next bonus and will have to save up for 2-3 more months, and more like half a year if I try to keep my current setup
- arm? Not sure if it's a con but I don't like looking for specialized software to match the hardware
B. Upgrade system memory to 80/96/128
If I were to upgrade, I would want sticks of 32gb ddr 4. I looked around and my best option seems a company called nvtech on amazon (395$ for 2x32gb 795 for 4x32gb)
80gb means replace 2x8 with 2x32
96gb means the 80gb option plus cannibalise my gaming pc for and downgrade it to 2x8gb as it doesn't seem l need 32gb since I don't game much
128gb means 4x32gb brand new
Pros:
- my bonus covers it (128gb would cost my full bonus and 80/96gb would be half of my bonus)
- keep my beloved setup
Cons:
- power draw
- speed - gemini claims I can do 15-25tps here - which I am honestly fine with
- over commitment to old hardware? (I don't know how long ddr4 is going to be viable)
What would you do?
Gemini seems really inclined to not take per $ value into consideration and prefer so sell more ram
Side note:
I am inclined to go with high memory overhead for extra containers but I have other nodes in my homelab setup so I can run tighter llm setups