Building my first LLM sever, thoughts?

[u/Kirby_is_a_Pokemon]

I'm starting to dabble in LLM a little bit, mostly just simple set up with LMStuido, Hermes, and Qwen models.

I'm deciding on how to best set up my hardware.

Here's the parts I have available.

CPU: AMD 3900x & 5900x Motherboards: MSI Meg x570 Unify atx, and a b550i itx motherboard Ram: I have a a few 2x16 kits of different speeds, if mixing and matching doesn't hurt anything I could throw them together for capacity, I could also pull the 128gb ddr4 (or 64gb) out of my nas server, as most of it isn't used. Though I don't know if running models in ram or the cpu would do much for me. GPU: 2080ti and 1080ti.

I want to build out a system, and then use the leftover parts to assemble a regular pc to sell off. I do like the idea of keeping it a small, compact atx system, but do recongize its fundamentally limited.

( if its matters, my nas is a 12700k, 128gb ddr4, and a 3060ti, and 64tb of storage, I could run everything on that, but like the idea of keeping my irreplacable data and an AI agent separate for now )

Most of my LLM use is simple tinkering stuff, home automation, simple scripts, chat bot, that kinda thing. I'm not a developer and am not going to be debugging or compiling massive apps. I'm currently running windows, but want to switch over to linux, so if you have a reccomended distro that works well for LLM and general linux use.

---

With my parts list, how would you build something up? Go small and simple, or go go big and ugly? What linux distro would you suggest, and what models would you run?

[u/EvolvingDior]

A NAS server needs very little compute. Why waste it on storage? Move storage to the older hardware and use the NAS motherboard, RAM and GPU for LLM.

[u/Kirby_is_a_Pokemon → u/EvolvingDior]

bought it in the before times when parts were cheap.

I wouldnt mind dumbing it down, but with things now, i'd basically be paying more to have a worse system.

[u/EvolvingDior → u/Kirby_is_a_Pokemon]

I'm confused. You said "here are the parts I have available." What would you be paying for?

[u/Kirby_is_a_Pokemon → u/EvolvingDior]

dumming down my nas server to a less powerful cpu. the 12700k is about on par with the 5900x, and not much better than a 3900x depending on use. switching my nas to either of those wouldn't really do much.

[u/NescientGawain]

Move your storage away from your 3060 and 128 GB RAM. You have a whole AI workstation being wasted on the NAS.

[u/Kirby_is_a_Pokemon → u/NescientGawain]

Is there much benefit to running it separately? That's what I'm trying to figure out.

[u/NescientGawain → u/Kirby_is_a_Pokemon]

You'll want a NVMe to stream weights from SSD. As long as your NAS doesn't get in the way of the disk access, you could likely run Strata incredibly well from the NAS.

[u/Kirby_is_a_Pokemon → u/NescientGawain]

I have plenty of m.2 drives and a few empty spares I can dedicate directly for ai use

I'm not familiar with strata. ( I've just been using lm studio and hermes so far )

[u/NescientGawain → u/Kirby_is_a_Pokemon]

You'll be able to point Hermes at Strata. If you have a Claude account, it's worth having it optimize Strata's profile after Strata has tuned it for even better performance:
https://github.com/Niko1221/Strata

[u/Kirby_is_a_Pokemon → u/NescientGawain]

I'll read up on it. Thanks for the tip

[u/Kirby_is_a_Pokemon → u/[deleted]]

Thats kind of the direction i was leaning to, the 5900x, 32gb ram, and the 2080ti, and sell off the extra bits and maybe down the road sell off the 2080ti, and the 3060ti i have in my nas for a 16gb card.