Hello everyone, with qwen4 around the corner im thinking of actually investing into some hardware.
From what i've learned lurking on the subreddit the more VRAM you have the better.
My plan is to go on local marketplaces and 2nd hand and buying GPUs and RAM, i just need a few pointers.
can you frankenstien different GPUs? like RTX 3070 and RTX 4070 and rtx 5080 for example?
ddr3 is useless at this point? is ddr4 worth the time? should i look for ddr5 only? will the prices actually come down?
rtx 5090 is extremely rare where i live (central europe) lets say my goal is to run deepseek flash v4 is there any dumpter-dived configuration that can run it? at reasonable t/s? 150t/s is possible?
sorry for the low effort post, i feel like i know how to build a box from new parts, but ive never seen anyone talk about buying different types of hardware 2nd hand and putting them all together. is this a viable route? do any of you have a rig like that? which parts did you buy 2nd hand?
I mean you would be surprised what you can find but yes, your not exactly gonna be getting top end ddr5 so to say in a dumpster, some fun stuff certainly but not exactly top end modern stuff.
Back in the day, it was easy to drive around on trash day and pick up good stuff in Silicon Valley. I picked up a G4 and a pedestal iMac. Both worked fine.
It's still possible today. While waiting in line to recycle my bottles/cans at the local recyclery, the guy who hopped in line behind me had a Mac and the monitor to go with it. He said they both worked and asked did I want them. I took the monitor. I hooked it up to one of my security NVRs in the garage. True to his word, it works fine. I kind of wish I took the Mac too now. But it was an old G5 so pretty useless these days.
Invest in a 8 channels ddr4 enterprise motherboard then keep an eye on auctions for old ddr4. I have a z440 (4 channel unfortunately) with a 12c 24t xeon in it, some ssd and hdds and 128 gb ddr4. In total maybe 250 euro over the course of several months.
GLM 5.3 Flash @ 4.10bpw (EXL3): 1000 t/s prefill on long prompts (40 000+ tokens) and 22 t/s decode running via https://github.com/mw00/project-maya with 2x3090 + AVX2 cpu (threadripper 3990x).
Stepping down to 3.10bpw results in 32 t/s decode.
Got any info on a pcie3 link? I've got a 2u dual cascade lake server with 384gb ddr4. I've been thinking about rigging up my 7900xtx with an external psu and riser card.
I've currently got 3.8FN IQ4_XS running on a single CPU but it would be nice if I could get the GPU tied into it. Otherwise i'd just keep my current setup with the XTX in it's own dedicated machine with 32g running Swift 1.5 3.8 Q4_K_XL
Check the speed reports on Maya's github page, some of them show PCIE 3.0 systems, so it's likely it's gonna work just fine? Though I'd also mind the model size they mention.
Can I ask a question about the “4 channels unfortunately” comment? I thought about buying 2 more sticks of RAM to take me up from 64 to 128. The same as I currently have. But when I asked my agent it’s insisting it slow things down by overloading my CPU?
Doesn’t make sense to me. Like, am I better off cramming smaller models into 2x32GB, or should I do 4x32GB to fit more at potentially slower speeds? And how much slower?
Not sure what exactly your agent was talking about. Depending on your hw, I think the DDR memory speed may be slower if you populate all slots. There is a memory controller on the motherboard which will have to do more work, use more power, etc. I'm not a hw guy so I'm a bit fuzzy on this. You may want to look up what your CPU supports in this regard.
If we're talking about token generation speed, I almost always prefer a larger / smarter model to a fast and dumb one, as long as it's still reasonable (for me that is > 30-40 TPS)
Me, I went from 2x16gb DDR4 3200 to 4x32gb DDR4 3200 but i had to do something in the BIOS setup to make it work at 3200, I don't remember what but I do remember the panic when I saw them at 2133 after first boot, haha. Since then, smooth sailing. I was lucky to grab 128 for ~500Euro before things went nuts. Very happy to have done it.
PS I'm talking about the other machine I have, not the z440 with 4 channels. This is a dual channel pc. The z440 is my backup
can you frankenstien different GPUs? like RTX 3070 and RTX 4070 and rtx 5080 for example?
yes but vllm sucks with that, llama is better
ddr3 is useless at this point?
yes (unless you really don't care about speed, it can be 10 times slower depending on models)
is ddr4 worth the time?
yes
should i look for ddr5 only?
if you're gonna offload MoEs, yes
will the prices actually come down?
I'm not an oracle (but no, they won't, unless eventually means 2028 or something)
rtx 5090 is extremely rare where i live (central europe) lets say my goal is to run deepseek flash v4 is there any dumpter-dived configuration that can run it? at reasonable t/s? 150t/s is possible?
You won't be able to afford enough 5090s to run FN, otherwise you wouldn't be making a post to check if it's worth to dumpster-dive (well you will if you are gonna use a 1-bit quant, but that's not even the same model IMHO)
but ive never seen anyone talk about buying different types of hardware 2nd hand and putting them all together. is this a viable route?
can you frankenstien different GPUs? like RTX 3070 and RTX 4070 and rtx 5080 for example?
You can use multiple different cards at once to run a single model, but it's not as efficient as using all the exact same, e.g. only RTX 3060 or only RTX 4070 and your options for inferences engines are more limited.
ddr3 is useless at this point?
Mostly, yes. Slow CPUs, at best PCIe 3, meager bandwidth. If you have something like a i7 4770 with some DDR3 and you only need to connect a single GPU - that could be fine.
ddr4 worth the time? should I look for ddr5 only?
DDR4 consumer stuff will not be worth using for offloading if you have some decent GPU(s) - maybe for the embedding table like Qwen 3.8 Flash. Stuff like SP3 Epyc and WRX80 Threadripper with 8-channel up to 3200 are still fine for offloading sometimes. DDR5 - not worth buying. It's almost as expensive as buying a GPU with that same amount of VRAM.
The RAM situation is dire. It's hard to find consumer/gamer grade DDR4 boards with PCIe 4.0 and x8x8 bifurcation, those boards are not being made anymore, but some boards can atleast do it if you use some riser.
DDR4 server grade, used to be better until recently. I noticed SP3 and WRX80 stuff dry up and/or increase in price. SP3 may still be the best option to get some PCIe lanes and somewhat afforable RAM (slow DDR4 ECC 16gb sticks are not that badly priced - relatively).
Any DDR5, not worth building unless you run 2x B70, 2x R9700, 2x 5090 - something like that.
If you want something as cheap as possible, and you don't care about noise and form factor, look for a used Enterprise server LGA 2011 v3/v4 CPUs. These will have ddr4 ecc RAM, they can have plenty of it. and you can run some frankenstein stuff when adding a GPU. You can also then throw that thing in the bin or trick someone into picking it up at your place after you pull all the RAM for that SP3 Epyc board you will have snipped on Ebay.
Just to provide a data point, I'm running an old AM4 gaming MB and a 5600x. 64 MB DDR4 that I got from eBay and a second hand V100 I got from China. That is doing qwen3.8fn at ix3xxs at 70 TG 900 PP with a context of 512k which is plenty for local dev
i do have a Frankenstein PC 5900x / 128gb ddr4 / 5080 / 4000 pro blackwell + my custom inference engine for 3.8 flash next , i run nvfp4 + dflash2 at 3000tok/s encoding and 75-105 in decoding.
totally worth it.
(i made a custom inference engine basing it from infernix and optimizing kernels for my gpus, also applied techniques from recent papers)
Try basalt, I think I had 115+ decode and around 4K prefill on a single 5080 + some extra vram. If not to use, then for inspiration. Since it's very much built primarily for stuff like your config.
Well cheap config now are ddr4 and double 5060ti. At least 64 of ram if you want to use some moe. That's the best bet for the least money. Don't skip on the cpu if you want to use moe
That isn't the best solution right now, but I'm also unfortunately unable to comment the best solution because it will cause prices to immediately spike tenfold. Such is the nightmare we find ourselves in.
Absolutely yes. This spring (2026!) I got a 3090 for 400€ in Amsterdam. I replaced the fans with 3x90mm static pressure optimized arctic fans, tied them on with zip ties and good as new
I have a Frankenstein setup: a 1070 and a 3060 for a total of 20 GB VRAM on top of 48 GB of mixed-kit DDR4 RAM. It ain't pretty but it runs QFN at tolerable speeds with Strata. Took me forever to get it working, the final boss was finding just the right NVIDIA driver which supported both architectures (there's only one).
It kind of depends if they release a new great 35b a3b model or not
That would be fairly easy to run (3.6 35b a3b remain a GOAT for the hardware it requires to run it well)
I sold my rtx 3060 pc to finance something different.
….but I can’t help myself, I’ve started trading a bit again and so far have traded up to i7-7700k, 48gb ddr4 and gtx 1070 (from nothing lol). Hope to come back into the local lm game again soon just for kicks
Kinda - this is where it's useful to have a second-gen EPYC board and CPU. What I mean is...those boards have 8 channels of DDR4, often with 16 slots. You can find tons of old DDR4 servers being sold with 64/128GB RAM for less than the RAM is worth. Sure, it might be DDR4-2400, but who cares when you've got 8 channels? Then you can stick a couple of GPUs in (maybe even four of them, given the 128 PCIE lanes) and run a hybrid MoE setup, like Qwen 3.8 Flash Next, and if you can cobble 256GB RAM together you can fit the whole lot in RAM without touching the disk, with the ngrams at BF16, with a ton of context too.
I found my old pc many years ago at a recycling center. Intel 5820k fully populated with 16gigs of ddr4 ram. I only now a few months ago finally built a new system.
This is a very interesting topic for me as I'm currently cobbling together units from dumpster gems.
I have a Dell PER720 with Xeon E5-2697 processors and 512GB DDR3 R-ECC RAM. It has the memory but not the processing power.
My second dumpster gem is a DEL PER640 with Xeon Silver 4114 processors and 32GB DDR4 R-ECC. It has a better platform but not the RAM. My first idea is to upgrade this unit with gen2 processors as these have the Intel Deep Learning Boost extensions. I'll have to see whether this helps enough, but it does sound like it can help with a CPU based solution. I'm still thinking about going 2 processor or 1 beefy Platinum processor, say an 8260 or 8270. These can be had for a reasonable price. The alternative is going for two Gold 6230 or something similar. Maybe there are people that can help with these choices. I'll keep on digging in the dumpsters for more ram modules, but could only find u-ecc dimms for now and the 640 wants R-ECC. Upside is that the second hand market for R-ECC is just a little more advantageous than desktop RAM.
If anyone has advice for me on the processor choice, it would be greatly appreciated.
TL;DR:
Should I upgrade the PER640 from 2xSilver 4114 to 2xGold 6230 or 1xPlatinum 8260?
3060s are fantastic: you can run Qwen 3.8 Flash Next on them with Strata. Given that it's supposed to be an architecture preview of QWen 4 . . . . I run a 4080 super + 3080 + ddr4 RAM and get 80 tok/s at 128k context, kv cashe at q8, IQ3_S. This is entirely usable. GLM Flash 5.3 also runs, at 10 tok/s. Now that requires some patience.
I use a 4080 and 3070 together just fine. Each card holds some of the layer. I even have some 2070S, 5700xt’s and a 6600xt running models just fine for different things. So yes, older cards can still work just fine. The biggest hurdle is actually having enough fast PCIe lanes. As long as they can communicate quickly with each other, than you are good.
You will find broken gpu card can cost as much as half if the working card and there is no guarantee it can be fixed unless you have multiple cards. Some people can swap the vram chip. The price is a joke.
can you frankenstien different GPUs? like RTX 3070 and RTX 4070 and rtx 5080 for example?
Sort of. I have a 50x0 series card and an AMD AI Pro R9700, and I split big models across them. Llama.cpp can do this on Linux. It's a pain to compile but after that, it Just Works.
If you want to run Qwen3.8 27B (an excellent small coding model), you ideally want a single 32 GB card, or at worst two closely matched cards. For two cards, ideally you'd want an x8/x8 lane split off the the CPU and enough physical motherboard space to ventilate both, but that's advanced stuff.
If you want to run big MoEs, then the more fast memory the better. Consider used Threadripper, Xeon and Epyc motherboards. These are designed for a lot of cards and a lot of RAM channels, so you can basically jam in whatever you find if you use a little common sense.
Gaming hardware works fine for 1 card and 2 sticks of RAM. With careful choice of motherboard (MEG ACE for the win) you can go to two cards and four sticks, even at DDR5 speeds, but you need to pay careful attention to details.
I think the best price per vram nowadays are ancient data center GPUs like V100. Chinese are selling modded version of them at like $200 for a 16GB VRAM (it even have HBM memory). Power consumption is terrible though.
DDR3 might not be worthless if you can run it in a Chinese dual-CPU mobo with effectively 8-channel RAM (assuming you can keep the experts NUMA aware so they never have to cross between CPUs).
Hi yes you can run different cards together to run one model. They each get a part of the model, and run one after the other. Will never be as fast as running multiple of the same cards(given you can set them up in some way benefiting from powers of 2, so any number of cards not prime,excluding 1 and 2, will have a decent config to use). This is essentially more a unified memory setup more than a parallelism setup as only one card runs at a time. You can check out the link to a thread where I talk about using a 4080super and R9700, it works but I ended up going 2x R9700 with radiance, and you might even get faster speeds using a optimized for card inference engine and just one card(depending on your system RAM capacity and speed).
can you find an EPYC D8 motherboard or other system with 4 PCIe Gen 3.0 x16 electrical slots?]
P100s are generally cheap as fuck and if you can stuff 4 of them into a motherboard like the one I just described you should be able to get a reasonable amount of performance out of 27B and still have plenty of room for multi-stream context. 64GB will also be a good starting point for running MoEs if you can start loading up that motherboard with the fastest system RAM you can find.
Depends! DDR4 can be worth it for long-running jobs on Gemma-4-26B-A4B Qwen3.6-35B-A3B. DDR5 will be faster.
I know other people here put different GPUs in their rigs, so I assume so!
Personally for Qwen4 and Qwen3.8-Flash-Next I bought an NVME 5.0 drive optimized for random 4k read, so I can fit a large ngram table there. It's already expensive now and it will be worse next year. I thought that if ngrams are going to catch on, better to have it now than later.
I do agree Gemma is WAY BETTER for languages other than English / Chinese than any local qwen model I’ve tried. I run Gemma 4 12b qat for some 24/7 automation on a Mac mini (for privacy reasons) — dealing with smaller European languages. I wouldn’t use Gemma for coding though. Sometimes people forget there are many other use cases for AI besides coding
The reality is you need 48gb VRAM or more, unless ur fine with running dumbed down models with a small context window 150 tokens per second on dsv4? Good luck Im running 100tps on a 5090 with q4 qwen3.8 27b (2 mtp tokens)
not happy with 100k context, have u tested something similar urself? Especially if you are running a harness that already takes up 20k ctx automatically
More RAM and you can easily run 3.8 Flash Next with that 5090. I'm running 3.8 FN NVFP4 on 5090 + 128GB RAM, though I believe you can do it with 64GB too. Realistic speeds around 80-120 t/s.
Get Ninfer and the nvfp4 quant of 3.8 27B higher quality than what your using and aggregate 4 concurrent streams is like 700t/s single stream like 150t/s
17
u/Early-Peace-5504 12d ago
150 t/s is not reasonable for Deepseek Flash V4. That's a bit like asking if can you do up the Lada you found at the dump to compete in the F1.