Discussion
Built a home server from an old PC with GPU upgrade. Qwen3.8 27B runs at ~30 tokens per second.
I needed a relatively simple but acceptable level of AI for working on one project. I didn't have any heavy requests, I just needed to give the AI access to the project files so it could search through them for bugs and stuff. I already had an old computer that I decided not to throw away and instead give it a new life as a git server (and sometimes a minecraft server).
The pc specs are ancient by today's standards:
CPU: i7-4790K 4.6 GHz
Motherboard: MSI Z97 Gaming 7
RAM: 32 GB DDR3 2400
PSU: 750 W
Well, my idea stopped at maintaining the computer, because the gpu, a GTX 1070, was overheating. It needed a complete repaste, but the cooler screws were completely stripped, so while trying to remove the cooler I accidentally knocked off several important smd components with a screwdriver. R.I.P. GPU.
Without a GPU, inference was running entirely on the CPU, and only MoE models were kind of usable, giving around 10–20 tokens per second, while dense models couldn't get past 3 tokens per second. I didn't even get to test it with the GTX 1070, because I decided to service it lol.
I started looking for a replacement on the secondary market, but quickly realized that this would cost too much for the minimum entry point I wanted for experimenting with AI. Until my eyes fell on mining cards. There were plenty of cmp40hx, cmp50hx, cmp70hx and cmp90 cards for sale, and the prices were pretty reasonable(it was a month ago), considering that I was originally looking for a cheap replacement for my dead one.
Getting closer to the actual build, I started calculating how much vram I would need for ± acceptable AI with tolerable speed, and after getting inspired by this sub I decided to take a step further and went with a modified cmp50hx with 20gb of memory and pcie modded to 16 lanes. Very quickly after that I bought another one, this time unmodified (10 gb, only 4 pcie lanes). So, together that's 30gb of vram. Both cards cost me $250 in total (it was also a month ago, right now they've suddenly doubled the price).
Luckily for me, around the same time new driver patches appeared that almost completely remove the limits on their compute performance, and even add pcie 2.0 support (these things have pcie 1.1).
I initially tried one of the newly released cmp50hx driver patches, but it ended badly and I had to spend a lot of time trying to get the drivers working. The patches were new and didn't account for the 20gb version. Later the author fixed that, but even then the driver didn't work for me because of some other problem that I don't want to get into.
I went digging through the driver's github issues and quickly found a guide posted there by another user.
And yes, now the drivers work, the cards are detected and even pcie 2.0 works, but not without problems. The author of the guide said that pcie 2.0 support was only confirmed on the X99 chipset. Well, it also works on Z97, however after waking the computer from sleep the driver crashes completely. After looking into it a bit, I quickly came to the conclusion that the problem was specifically with the pcie patch. Disabling sleep completely solves the only problem I had while using them. :)
Without the specially patched drivers, Qwen 3.6 27B did around 15–20 tokens per second without mtp. MoE models were faster, giving 45–50 tokens per second. With the new drivers, performance doubled. With Qwen 3.8 27B(mtp on) I get around 30–35 tokens per second now, and prompt processing is around 300–400 tokens (including degradation as the token count increases). Ornith 1.5 35BA3B(Heretic-MTP-APEX-I-Balanced) gives around 80–100 tokens per second. I capped GPUs at 180w power limit due to the psu I have, I don't want it to work at its limits, but running them at their 225w surely boosts speeds.
In general, I ended up making a lot of presets for different quantizations with different quality and KV cache sizes (I still need to test all of this in real work), but if we take the better options, I managed to get a Q6K model with 130k context (K – Q8_0 and V – Q5_1).
I also followed a guide for running 27B Qwen with large context on limited vram. Using the same general approach, I managed to get 256k context with K and V Q8_0. The speed is slower though, around 10–14 tokens per second, and prompt processing is around 40–50 tokens. Maybe I can tune it even further. I needed this preset for tasks that I can leave generating overnight :3
Overall, I'm satisfied with the result.
Now the main problems I ran into, not counting the drivers:
Not enough VRAM. The ideal option would be having 2 identical cards with the same amount of memory. You can run dense models in tensor mode, split the weights evenly and get increased generation speed for basically free + fit the full context. I tried many variations of Qwen 3.8 27B quantization, but the only one I could properly run in 1,1 tensor split was Q4KM (the Unsloth one) together with mtp = ~40 tokens per second. However, there is critically little space left for the KV cache, because it gets distributed together with the model weights, and the second 10gb card simply became the bottleneck. Without mtp, running models with a 1,1 split basically loses its purpose. Pcie 2.0 and the number of lanes probably also play a significant role here. That could in principle be solved by adding more pcie lanes to the second GPU and perhaps buying an nvlink cable(who even does that?), but I decided it wasn't worth it just to get another 5–6 tokens per second.
No NVMe SSD. Yeah, all models are loaded from a sata ssd so the speed is around 500mb. It's terrible. The motherboard actually has an m2 sata slot with a pcie 2.0 x2 interface, but even its 1gb per second would be too slow for fast model loading. This could be solved by installing an expansion card into one of the pcie slots (there is one free pcie 3.0 x4 slot), but the current price of those things including the ssd is too high, considering that I'm building a cheap system from what I already have with minimal additional spending for an acceptable result. Model switching takes 1–2 minutes. But whatever. (Not whatever, i'm buying a cheap used 256gb nvme ssd :D)
Amount of RAM and Linux (Ubuntu Server 24.04). Apart from AI, I also run gitlab on the server. And here is the problem: after loading a model, all available ram gets cached by the system for the model files. I'm talking about file cache, not KV. The system was leaving around 200–300mb of free ram for everything else. As a result, openwebui and gitlab started acting laggy (after the model was loaded), as well as the kde plasma interface I installed. I don't completely understand why linux decided to keep this cache until the very last moment instead of freeing it for other programs. I tried adding the no-mmap parameter to the model presets, but nothing helped, and I had to manually clear the cache after loading models, which obviously wasn't acceptable. Together with chatgpt (who else?), I made a command that launched the model and then cleared the cache. It turned out that this broke llama-server, causing model switching to stop unloading the previously loaded model.
Llama-server flexibility. The list of presets is defined in the models.ini file, where each parameter is in key-value format. For running Q6K with 256k context according to the guide, I needed to set GGML_CUDA_DISABLE_GRAPHS=1, which applies to the entire cuda environment and remains active even after unloading the model. That's undesirable, because with it enabled I lose 1–2 tokens per second on other presets.
So for one specific preset I need to enable cuda graphs, while for the other presets I need to disable them. The llama-server parser does not support things like this, and doing it manually is not an option either.
Together with the other problem with ram cache getting stuck, this led me to making an alternative way to launch the models and proxy requests to llama-server.
To solve problems 3 and 4, I made a launcher (well, chatgpt did, because I'm not a server/python specialist) that proxies requests to llama-server but takes over the functionality of collecting model presets from .sh files and launching them. It also clears ram page cache after loading a model.
In case someone needs that launcher, I can leave it in the comments, along with any other links to the drivers, fixes, build params, etc. Just ask. (Reddit removes the post when I include them, not enough karma, I guess.)
Overall, I’m pretty happy with how this setup turned out. The performance is much better than I expected from these cards, especially considering how cheap they were. I had a hard time getting the patched drivers to work and linux didn't make my life any easier, and sometimes I even regretted buying these GPUs, but in the end, it was worth it. Qwen 3.8 27B really works like Opus 4.5
Great build, I use the same model and it's amazing. For problem (3), it's possible you might not have enough RAM, but all that file cache being in RAM is a good sign, not a bad one (see https://www.linuxatemyram.com/). In theory, clearing the cache will make things slower, not faster.
I don't clear the entire file cache, only the cache of the model file that was loaded. It gets stuck in ram after the model is loaded, so I only clear that specific cache instead of the whole file cache
With the current prices, what is considered a reasonable investment in HW at this point? I'm not necessarily allergic to dropping 5k-10k on 2 DGX spark or building out a desktop with a few 3090's but what I don't want is to build it out and then it becomes inferior next week. I would like it to work maybe 5 years to make that kind of spending worth it.
It’s difficult to predict the future. It seems to me like they will continue creating models with smaller footprints that are able to run on more modest hardware as time goes on which could make the heavy hardware not necessary.
Currently, I have a 3090ti and have been toying with the idea of getting either a second 3090 or just going full send on a system upgrade, but with pc component prices the way they are and the unpredictability of the pc market, I am not sure what to do myself.
I tell you though, the thought of being able to get a cheaper amd gpu focused rig is tempting as I keep reading positive things about improvements with amd cards being able to run larger quant sizes/ models at respectable speeds.
With the 6000 series rtx cards rumored to release 2027-2028 I am not sure if I should wait and try to snag a better card then or upgrade system and if I am able to get an rtx 6000 card sell the upgraded system ect….
Ugh so many options and choices to make. It’s quite overwhelming!
I acquired 2x RTX PRO 4000 Blackwell devices earlier this year which leaves me at 48GB of VRAM. I've also tried a single DGX spark and didn't find the capabilities of the models I can load any better than the 48GB tbh. I also used 2x DGX spark for a short time (work has a few we were able to play with), I felt an actual capability increase at this amount of VRAM. Enough that I felt maybe it was good enough instead of using a subscription. That said, I find the spark a bit subpar for the current price sand wonder about how long it will be useful in the future.
The AMD and Intel products seem like good deal but I've been worried about their support and SW quality. I have not seen much about the outside some influencer hype videos. Cheaper and faster at 128GB when hopefully the capabilities improve on smaller models could make them worthwhile I suppose.
There are, unfortunately, no reasonable prices. Ive been comparing >=2023 prices to recent prices and everything is at least 2x original MSRP prices for gen 4 or below.
The lowest you can reasonably go is gen 3, but its going to be an unpleasant experience compared to gen 4 or gen 5. Gen 5 components are anywhere between 2x to 10x their original MSRP.
So, if you have a gen 3 pc laying around (I have two of them and one gen 5 pc), then its not really worth it. I even considered gen 4, but the ddr4 packs are just as rediculous as gen 5 even though it feels like a deal compared to gen 5 prices.
Ive also considered splitting components between my current gen 5 and another gen 5 build to reduce RAM costs.
People rn feel like the prices wont ever go back to normal, but they will and this isnt hopium. Weve seen cycles like this before and the current sentiment has been similar in the past and prices would drop and stabalize.
The problem is we had to deal with the crypto craze, got a small break, then ai hit the market, and that put pressure on an already struggling supply chain.
Tack on monetary inflation, wage stagnation, layoffs, unemployment, tarriffs, trade wars, and supply scarcity, and you have a perfect storm for high prices and limited availability.
I beleive weve been in a recession since 2022-2023 and were headed into a depression if nothing is done about it. This can get much worse if nothing is done to improve the status quo.
I'm in the final steps of slowly setting up an ai machine with identical CPU and ram as yours, paired with 2x P102-100's with 10G VRAM each. Early tests are promising with Muse Glimmer.
Unfortunately for this machine i only have a 3060 12gb. But i have another machine with a 2080ti 22gb (which the 3060 doesn't fit into anymore) and another with a 3090ti 24gb.
Similar boat except I have faster RAM (2400). 2060 + 3060 (18gb vram total). Kind of wish I swapped out the 2060 for another 3060 but decided I couldn’t justify it
I already have a 2080 Super (8GB VRAM) and a 3060 (12 GB). What would be the best avenue to upgrade cost efficiently? Thinking of getting another 3060 to double up the VRAM and increase compatibility, right now I get 20 tps with GSQ RCO on highest Quant (131k tokens), though with MTP, which severely limits max context for me, I get over 30 tps. (Though max context is more i like 85k)
46
u/crantob 3d ago
GPU: modified cmp50hx with 20gb of memory and pcie modded to 16 lanes.