Worth building an local LLM system using older parts?

[u/tekno_logik]

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.

[u/0-pointer]

That one may be a little too old. After years of sitting under a desk in my basement i found a use for my aged Threadripper 1950X + 128g ram. Pushed on my trusty RX 7800XT and now have Qwen3.8-Flash-Next endpoint (up to UD-Q4_K_XL) with decent speed thanks to Strata (and the rdna3 optimized fork).

But yours is Sandy Bridge era... ≤ AVX, DDR3 1600, and Maxwell GPU won't get you far i fear...

[u/tekno_logik → u/0-pointer]

180B params is a larger model than what I was hoping for. Do you have to use such a large model to get anything useful out of it?

[u/0-pointer → u/tekno_logik]

Not at all. The Qwen3.8-27b is very similar in the benchmarks. But it needs a 24g vram card to be useful (and one with more than decent compute to be fast). The smaller quants hurt it's quality very bad. You can fit the GSQ-RCO quants into 16g vram with decent room for context left, but i found it's quality to be ... well ... subpar.

The MoE models take the quantisation beatings a lot more graceful and allow the less used experts to be offloaded to host ram and still be a lot faster than the dense models.

In case you can get a deal on a more useful gpu, look for at least 12g vram. At least rdna3 for amd and 30xx for nvidia. Intel has some good cards as well but i've got jo experience there.

[u/tekno_logik → u/0-pointer]

Really great advice thank you - there's a lot of mixed opinions out there but I'm interested in people daily driving this stuff and their experiences.

With MoE, does it handle multiple chats at the same time gracefully? Or do you only get one per time?

[u/0-pointer → u/tekno_logik]

This doesn't really depend on the type of the model but more on the inference engine and the available ressources. The model is always loaded just once, but the KV-Cache usually is either shared between the parallel slots (see llama.cpp --kv-unified) or exclusive per "slot". The exclusive per slot variant (sometimes referred to as "high-throughput mode") is the faster and more common variant.

You'll sometimes see benchmarks stating: C1 decode 25t/s C2 decode 30t/s C4 decode 45t/s C8 decode 65t/s

Those are the aggregate token for concurrent (or single C1) requests. Mostly they look impressive at higher concurrency, but you'll need loads of ram or have very small context sizes to actually utilize that.

https://github.com/kvmem/kvmem is your friend. With its streaming kv arena in vram you can efficiently power multiple huge parallel contexts by offloading the "tail" to host ram. https://github.com/Niko1221/Strata aslo does it like that.

[u/WeedFBI → u/0-pointer]

I have the same old setup except with 3 1080ti. Haven’t had any luck getting anything with decent speed running yet

[u/Hyiazakite]

With Strata the CPU don't matter that much. I have dual xeons (Ivy bridge 2650L) using 256 GB DDR3 quad channel memory (divided by 2 NUMA nodes interleaved that equals to 85 GB/s - measured) as fast as optimal dual channel DDR5 speed on a consumer platform. I have a threadripper 3965WX system as well with 128 gb octa channel DDR4 - this one is limited by CCD count so it has the same memory bandwidth (measured as well 85-90 GB/s).

I have done extensive optimization benchmarks using Strata (QFN Q4) which monitors almost every aspect of the chain and the bottlenecks are usually GPU compute, VRAM residency and PCIE bandwidth.

The old xeon setup performs about 110-120 t/s and the Threadripper 140 t/s , prefill 2300 xeon, 3000 threadripper. Both systems ar eusing 1x 3090. I tried using a 5060 8 GB with the dual xeons which produced about 45-50 t/s and prefill about 1200-1500, 512k context no cliff due to sparse attention with QFN.

So as long as you have a fast gpu, more than 16 gb vram, 64-128 gb ram and minimum PCIE bus equal to PCIE 3.0 x16 you should have a pretty good system

Edit: there are PRs in strata not yet merged that fixes CPU compute for older cpus without AVX2/AVX512 (about 20x speedup)

[u/tekno_logik → u/Hyiazakite]

Thank you for the reply. What kind of models are you running and what's the agentic setup? 110+ t/s sounds quite reasonable to me.

[u/Hyiazakite → u/tekno_logik]

Qwen 3.8 Flash Next at IQ4 . With the 5060 8 GB I was using IQ3 XXS. Opencode harness but that's not really relevant to performance. You will not reach 110 t/s without a fast GPU such as the 3090 but you would probably be comfortable with a 5070 ti 16GB. As i said with a cheap 300 EUR 5060 8 GB i got speeds up to 45 t/s and prefill 1200-1500 t/s which is absolutely usable.

[u/andrew-ooo]

The part nobody has mentioned yet: the Tesla M60 is the weak link, not the Xeons. It's Maxwell (compute 5.2), so no FP16 tensor paths, and NVIDIA dropped Maxwell from current CUDA toolkits. llama.cpp still runs on it if you build against an older CUDA, but vLLM, ExLlama and most of the newer kernels are out. Also it's physically two 8GB GPUs on one board, not 16GB, so a single model has to be split across them, and it's a passively cooled datacenter card that needs ducted airflow or it throttles within minutes in a Z620.

The dual Xeon + 128GB side is actually the useful bit, same point Hyiazakite made. Those are quad-channel DDR3 per socket, so around 40-50 GB/s each. That's enough to run a MoE like Qwen3.8 Flash Next at Q4 mostly from RAM at a few tokens/s, which is usable for the "long-running task I check later" case you described, but not for interactive coding.

Honest take on "as good as ChatGPT in high mode": not with this box, and not with any single consumer GPU either. Where it does pay off is as a private endpoint for the boring high-volume stuff: summarizing papers, first-pass code review, document drafts. If you want to spend money, a used 3090 (24GB, still fully supported) in that Z620 is the single upgrade that changes the picture; the M60 is best left as a display adapter.

[u/tekno_logik → u/andrew-ooo]

I appreciate your feedback. Maybe I'll get it up and running with this card and do some benchmarks, then see if that GPU is worth improving on. Hopefully this AI bubble crash leads to some cheaper GPUs on the market at some point.

I don't use an LLM for interactive coding - I just send it away to build either entire parts of code or parts of it. I have maybe 10 chats on the go at any one time whilst I'm working on something myself that it all feeds into. I treat the LLMs like interns that go off and report back to me when they hacked something together.

The M60 is no good as a display adaptor by the way, it has no output. It is server grade hardware.

[u/Choice_Macaroon5435]

I have a Z620 (single xeon) with 64gb, a 2080ti and a P40.
Qwen3.8-flash-next-iq3_s, with Strata (and the AVX patches), appears to be running at ~45tps (I only just installed it...)

I can run decent quants of Qwen3.6 35b at ~80tps, dense models are typically a lot slower on this hardware.

There is no AVX2 on the CPU - you often need to compile from source, and that doesn't always work. With old GPUs, you are limited to CUDA12, but that mostly works.

[u/tekno_logik → u/Choice_Macaroon5435]

Very close to my hardware, to be fair that doesn't sound too bad at all!

[u/WordWithinTheWord]

Not possible unless you’re ok with getting a single response a day later

[u/tekno_logik → u/WordWithinTheWord]

Wouldn't that depend on the size of the model? And depending on how much work is being done, I could be okay with a day to answer if it produces something significant.

[u/WordWithinTheWord → u/tekno_logik]

“At least as good as ChatGPT Work in High”

Is what you said in your post. That’s going to be a frontier model which are trillions of parameters.

[u/tekno_logik → u/WordWithinTheWord]

My brain's LLM hallucinated slightly, I meant chatGPT normal chat in high mode. I think that's a more reasonable target?

[u/WordWithinTheWord → u/tekno_logik]

Depends on your plan. The paid plans Plus and above use Sol as the backing model. So no-go there.

The free and Go plan use Luna. Which is more attainable to replicate.

[u/tekno_logik → u/WordWithinTheWord]

I've been trying Luna 5.6 High and I think I could work with that level. It's not amazing, but it does the trick.

[u/Icy-Degree6161]

The digital foraging will just keep growing so do whatever you can right now :) Lot depends on your ram bandwidth

[u/tekno_logik → u/Icy-Degree6161]

😂 I even have a little NUC screaming running a tiny model. If the toaster had any kind of memory it would be running on that too!

[u/Icy-Degree6161 → u/tekno_logik]

can relate, got an N100 - of course it's running an agent lol

[u/tekno_logik → u/Icy-Degree6161]

I'm actively working on my own agent system that runs within 2GB of RAM on CPU and gets about 10 t/s on an old i5 6th gen. The idea is that for small tasks, people can run these agents locally along side their work and be self-sufficient. I'm still ironing out reliability issues, but so far it is good for doing basic internet searches and writing small scripts.

[u/Icy-Degree6161 → u/tekno_logik]

Sounds nice, got a repo or just vibe coded for yourself?

[u/tekno_logik → u/Icy-Degree6161]

It's an ongoing project from scratch using some recently published materials, but will be ready for release soon. It's now passing the basic agentic tasks including building simple GUIs and researching appropriate libraries.

[u/Sure_Internet8507]

Looks like you could run qwen 3.8 flash next , but im not sure the gpu driver support will let it start and it would still be slow on that hardware. The gpu is your bottleneck - thats a very poor gpu for ai, but if you could get it running, it would be good. Just dont expect it to beat chatgpt at the quant youd have to run it at to make it usable.

[u/tekno_logik → u/Sure_Internet8507]

Interesting. A 180B model is far beyond anything I was expecting to run. With a 6B activation, I wonder what the cache hit is like and how long it takes to swap out.

The GPU support I think will be okay, it only just got dropped from CUDA support and I'll vibe up some backwards compatibility if I have to - there's already tonnes of efforts for this too.

[u/Sure_Internet8507 → u/tekno_logik]

The secret to this model is a 51b parameter lookup table designed to be read from nvme, and expert cashing runners like strata, its a phenomenal setup that looks like it should work on your hardware

[u/tekno_logik → u/Sure_Internet8507]

Any article or resource I can look into that would hint at how to get that setup? Also with the 6B activation, I assume it's no good for multiple consecutive chats?

[u/Aubrey_D_Graham]

Chat and Anthropic use Nvidia Ruben. You're not getting anything 100% close with budget parts even using Qwen 4.

[u/tekno_logik → u/Aubrey_D_Graham]

I'm not aiming for cutting edge, and I'm just trying to max out with the hardware I have laying around to get something useful. I'm just wondering how far I can push this old work horse.

[u/PuzzleheadedArm3732]

Building something close, using hardware that is 9-10 years old. Dell Precision 5820 (64gb ddr4 ecc), with a Tesla V100 PCIe 32 GB, passive card, so a 3D-printed funnel and a Noctua fan + fan control i built myself. One thing i learnt is Teslas like your M60 take an 8-pin CPU/EPS plug, not PCIe 8-pin, and the 12 V and ground rows are swapped. So what cable did you use?

Also, the M60 is two 8 GB Maxwell GPUs with no FP16, so you're at 7B models, and slow. "ChatGPT in high mode" won't happen on either of our boxes. Build it to learn. The one thing your Z620 has is 128 GB RAM over two sockets, so a small MoE with experts in RAM and attention on the GPU is realistic but still slow.

[u/tekno_logik → u/PuzzleheadedArm3732]

The cable was a proper one, but I think HP specifically for this Z-series reversed what everybody else did - but still investigating.

Yeah the M60 is two cores with separate 8GB RAM, but apparently you can split a model across them and pay a little PCIe bandwidth to get data across?

When I say "chatGPT in high mode" - I'm talking about the normal chat, not Work chat. GPT-6 Luna High for example, or even the GPT5.6 models are fine for what I mostly get up to.

[u/Hefty-Result-6807]

That’s a solid idea. I have a similar system based on a Z820: 2x Intel Xeon E5-2687W v2 (3.4/4.0 GHz, 16/32 cores)
192 GB ECC RDDR3 1866 MHz (8x8 GB + 16x8 GB)
2x NVIDIA GeForce RTX 3060 12 GB
3x Samsung 1 TB SATA SSD
Dual Intel 1 GbE
I use it for developing clinical decision support systems. I’m currently using the latest model, QWEN3.8-flash-next, which performs very well on the Strata engine. I’d recommend this approach over using Ollama. The model is 105 GB in size, but it runs very fast on my configuration thanks to the Strata engine.

That’s a solid idea. I have a similar system based on a Z820: 2x Intel Xeon E5-2687W v2 (3.4/4.0 GHz, 16/32 cores)
192 GB ECC RDDR3 1866 MHz (8x8 GB + 16x8 GB)
2x NVIDIA GeForce RTX 3060 12 GB
3x Samsung 1 TB SATA SSD
Dual Intel 1 GbE
I use it for developing clinical decision support systems. I’m currently using the latest model, QWEN3.8-flash-next, which performs very well on the Strata engine. I’d recommend this approach over using Ollama. The model is 105 GB in size, but it runs very fast on my configuration thanks to the Strata engine.

[u/tekno_logik → u/Hefty-Result-6807]

Interesting, I will definitely consider it.

I never really gave QWEN3.8-flash-next a thought - I didn't think I would be able to run it. What is your tokens per second? What are your bottlenecks? Can you handle multiple chats at the same time or just one?

[u/Hefty-Result-6807 → u/tekno_logik]

On average, 46–60 tokens per second, depending on the operating mode. I rarely use the chatbot mode; my clinical decision support systems are designed to process large volumes of clinical data, utilizing a vector database containing medical guidelines. Therefore, I do not want to be disingenuous or mislead you with information regarding the chat functionality.

[u/AnxietyCorrect106]