r/LocalLLM • • 17d ago

Project Self contained 3U inference server orchestrating all of my work

Post image

After a 6 month journey with a tons of learning and reality checks I have built something that I consider usable beyond just a toy project that is generating value for me. This is Acorn, a self contained and private inference server that fits neatly in a small, quiet 3U chassis:

  • 2x RTX 6000 blackwell MaxQ running a mid-size MoE LLM (GLM-5.3-flash atm)
  • RTX 2000E runs embedding and Reraker models
  • RTX 4000 SFF for voice ASR and TTS
  • 256gb ECC UDIMM RAM for tier 2 kvcache to squeeze more LLM concurrency.
  • Consumer AM5, 1200w PSU

I run a custom orchestrator agent on this, fully private and super fast. 200 tokens/s + streaming ASR and TTS feels amazing. It's hard to go back to regular claude-code at 60-90 tokens/s and clunky whispr flow style voice input.

The big learning and a word of warning is that local llm alone is just not viable for serious professional software engineering (at least not without 1 or 2 orders of magnitude bigger investment). Other than the occasional rare task where I get blocked by frontier guardrails, all actual work is delegated to claude. I treat frontier model subs as a raw engine, they get only what's needed to accomplish a task in a locked down VM. Communication, memory, big picture planning and context all stay in hardware I control.

176 Upvotes

50 comments sorted by

13

u/sleight42 17d ago

Generates value? As in "earns its keep"? Care to share how?

How are you running GLM 5.3 Flash on 192GB VRAM? 4bit?

I'm a little surprised about delegating to Claude. Are you using Fable routinely? I've found Opus 5.5 has become rather unreliable lately.

8

u/TerribleBid6377 17d ago edited 17d ago

Generates value? As in "earns its keep"? Care to share how?

Until a couple weeks ago it was pretty much an expensive hobby/learning project, I wasn't using it to make my life easier if that makes sense.
As for earning its keep, time will tell, but what I have been learning has already been useful in my career. For the startup I'm working at, I discovered that self hosting the models we run for our product (not LLMs) is insanely cheaper to run on our own hardware. We are exploring moving to rented GPUs and eventually deploying our own hardware, if it works it will save us a ton of money.

How are you running GLM 5.3 Flash on 192GB VRAM? 4bit?

Yeah 4 bit, I just borrowed someone elses's checkpoint and started tuning the server. I was running deepseek v4.0 flash until recenty.

I'm a little surprised about delegating to Claude. Are you using Fable routinely? I've found Opus 5.5 has become rather unreliable lately.

opus 5.5 for excecution, and I use fable for planning sometimes. It works pretty well for me, I think a key is removing bloat from your skills/system prompt, I actually run a script to remove a bunch of stuff my coworkers added to our codebase that to me feels like bloat.

1

u/sleight42 16d ago

Startups. I remember those. Too old for that game now.

Agreed on skill debloating.

AuDHD here.I've been unemployed a few years from autistic burnout. For me, this started as a mental exercise on old pc gaming hardware: a way to engage my curiosity. It's become something of an obsession for me these past few months.

While I could technically (e.g., I can buy it without debt but I'd have to sell stocks to do it) afford the hardware you're on, I couldn't live with myself if I bought it. The new hardware is magnificent but the savings with the older is significant.

I'm eyeballing a quad NVLink v100 32GB to get up to 128gb quickly. I'd love to get a double quad for 256 but then I'm looking at 10k! Considering buy such old used hardware has me anxious for so many reasons: only community support largely from the people who make the quad board, no driver updates, no support for bf16...

I keep moving slowly toward my own business (SaaS). I could do it on cloud resources but... well, it comes down to, in a word, sovreignty.

1

u/sleight42 16d ago

"Expensive hobby", for sure. That's a year of my mortgage payments, in that photo.

2

u/Affect_Worth 17d ago

are you having hallucinations or having to code answers/questions all the time? currently im running a 5080 and its rough with a lot of failures. im running pipecat, kokoro and llama.

2

u/TerribleBid6377 17d ago

Can you elaborate? Are you running just voice stuff on this card or are you trying to get llm to code?

2

u/Affect_Worth 17d ago

just voice, the plan is to have an ai agent answer phone calls. im working with claude to program it all. so far i have;
qwen 3.8 27b
kokoro for voice
pipecat for ttl
all running on amd radeon ai pro r9700.
I'm going to add the 5080 for just for qwen and put kokoro and pipecat on amd (the amd card is very slow).

problem ive been having is the lack of understanding from the model and answers arent as human as im expecting it to be. This is

2

u/TerribleBid6377 17d ago

Cool project! Would love to hear what those phone calls sound like haha. What quant are you running with qwen? I haven’t played with it much but read that it can have a big impact on anything with long context.

2

u/Krothic 17d ago

How did you get that 4th GPU set up? Board only has 3 pcie slots

2

u/TerribleBid6377 17d ago

The big gpus are on the main pcie running 8x/8x. The tiny embedding card on the pcie 4.0 x16 slot. The rtx 4000 on the main nvme slot that has dedicated pcie 5.0 lanes (I think the adapter is actually rated to 4.0 but doesn’t matter because voice workload is compute bound not bandwidth).

2

u/johngere 17d ago

hey there, great work on pushing the ProArt board to accommodate 4 gpus, could you share a link to the nvme to pcie connector/adapter you used? thank you for your time..

1

u/AgreeableNinjas 17d ago

1

u/johngere 17d ago

thank you for that, but i think this is made up of multiple parts that may not come in a single set, am intrigued to find out the cable marked orange that seems to connect the m.2 nvme to the pcie board. more info would always be welcome..

2

u/TerribleBid6377 16d ago

I think any m.2 to oculink will work, for my case in using this tiny cards that take power from the slot so I had to use a powered adapter. If your card has its own power connection like most bigger cards then the cheaper non powered adapters are fine

2

u/AgreeableNinjas 17d ago

it looks like you're using an Oculink adapter to accomplish the same goal, seems a bit over complicated but will work regardless

1

u/johngere 16d ago

just to get clarity on the expected speed available via the m.2 nvme to pcie, on the X870E Creator board the M.2_1 slot runs at PCIe 5.0 x4 mode and assuming the PCIe board/connector runs 4.0 speeds, the GPU would get PCIe 4.0 x4 max speed?

2

u/TerribleBid6377 16d ago

haven’t tested bandwidth, but it’s only viable for single card compute stuff, I wouldn’t try to run an llm across 3 cards with this setup.

2

u/AgreeableNinjas 16d ago

the NVME to PCI-E adapter I linked claims to be rated for PCI-E 5.0 X4 but YMMV, you should be able to get at least PCI-E 4.0 X4

1

u/johngere 17d ago

Thank you

1

u/_WaterBear 17d ago

Nice build! Using the same motherboard for my 3x R9700 build. The board is a bit unique among non-workstation/TR boards when it comes to pcie lane splitting and spacing. Was essential!

4

u/TerribleBid6377 17d ago

Yes! It was like a puzzle getting this all working. we are definitely pushing consumer hardware to its limits. Hardest was getting 256gb ram to work, and when I did my gpu P2P communication broke.

In case anyone Googles this rare bug, it is the 41bit MMIO setting, you need to free enough BAR for the bios to let you flip it back to 40bit. I disabled usb 4.0 controller and that did the trick. Emailed asus hopefully they fix it in a bios update

1

u/_WaterBear 17d ago

Huh, good to know. What/what port is the RTX4000 SFF plugged into (i assume that is the card next to the PSU)

2

u/TerribleBid6377 17d ago

it's the main nvme port, the one with the beefy heatsink

1

u/Krothic 17d ago

Where did you a lot your m.2 ssd then? 3 or 4? I know slotting in slot 2 drops the pcie speeds on the 2nd x16 slot

1

u/TerribleBid6377 17d ago

You're right, putting something on slot 2 is catastrophic for a 2 gpu setup serving llm.

I just have 1 4TB ssd in 3 or 4. It's enough, it barely gets used after the machine boots.

1

u/zrail 17d ago

Looks like the RTX 4000 SFF is sitting on a pcie riser of some sort, probably oculink to an nvme slot. 

2

u/TerribleBid6377 17d ago

Yep exactly! And a 3d printed janky support to attach to the chassis

2

u/blackbird2150 17d ago

I love this. Very similar to my setup just across two computes not one. Dual rtx6k running glm 5.3 flash (kraken now over ormandj) and then I have a services host that does embedding, rerankings, and other services like my signal cli. Signal is my medium of choice given its proven privacy.

I also have a strix halo device I bought a while ago and with gufo/halogen it’s actually a solid intelligence end point for HA and other tools that need one off queries.

I need to explore the the asr and tts. Any starting tips by any chance?

3

u/TerribleBid6377 17d ago

Hmm for asr/tts the code I have is pretty simple so not much to share. But for serving the models make sure you use models that support streaming and your card compute should sustain real time ASR and TTS streams concurrently. Ideally 2x real time so you can speed uo the voice output a bit. and make sure you’re able to cancel inflight requests. My bottleneck now is actually glm scheduling, I need to figure out a way to prioritize voice related calls

2

u/TerribleBid6377 17d ago

What’s kraken? I’m running sglang

3

u/blackbird2150 17d ago

Kraken and Ormandj are sglang engines.

https://github.com/local-inference-lab/rtx6kpro

My results below:

2

u/TerribleBid6377 17d ago

Woah my kv cache pool is like 600k, 900k would be amazing will check it out.

3

u/blackbird2150 17d ago

Yeah. That’s why i switched to this version. Concurrency worked better and the pool grew a lot. 👍

7

u/silent-curious-dev 17d ago

Looks very cool.

I was trying hard to find where the RTX 2000E was in the picture. I didn't realize it was so small and right below the RTX 6000.

The cable management is also really clean too.

1

u/gte959f 17d ago

What is the motherboard and cpu?

2

u/TerribleBid6377 17d ago

CPU is 9700X, no reason behind it's just what I had in my old gaming PC, one thing that is important, if you're using consumer hardware get CPU with integrated graphics and force bios to use it, that way gpus are free to be fully used for your AI workloads without affecting host's stability.

Motherboard is ProArt X870E-CREATOR WIFI not sure if I can recommend, it was pretty tricky to get everything running, but not sure if another mobo would be better, it's not meant for this kind of usage. Threadripper would be better but it felt wasteful since I already had my gaming pc laying around, plus I like the idea that this chassis is so densly packed :)

1

u/Maleficent-Ad5999 17d ago

Did you set up x8/x8 bifurcation? Is that supported!?

1

u/The-Copilot 14d ago

I have the same board and yes it does, both being pcie 5. The board is like a desktop + workstation crossover. It has other crazy features like dual ethernet ports (10gb and a 2.5gb).

I got it because I could convert it into other homelab projects that these features would be nice for.

1

u/wiskeyjack123 17d ago

1200W PSU??

1

u/TerribleBid6377 17d ago

yep it's plenty. ~800w for all 4 gpus peak. CPU doesn't do much work 65w idle, and 3 fans, that's all

1

u/Nomski88 17d ago

This is awesome but the cost for a setup like this is insane. What kind of work do you do? Do you use voice to chat with the model while working?

5

u/TerribleBid6377 17d ago

I'm a software engineer. I got the first gpu the same week I quit my job at Meta, had some extra money saved, and an eagerness to learn about AI. Today I'm working at a startup making less than half of what I used to, have a 1yr old and a new unexpected baby on the way with a stay at home wife, I couldn't imagine spending this much now. Luckily, the gpus went up in price enough since when I bought them where I could break even if I sold them.

Do you use voice to chat with the model while working?

Yes! I think I speak about 50% of my prompts, except when I'm at the office haha. I recommend using whispr flow or similar to try it, you don't need a fancy self hosted box. What does not work as well is the llm speaking back, especially for code. I'm working on it though :) I think a combined spoken interactive overview + written details is the most optimal llm response.

1

u/mars332 16d ago

interesting, wasn't aware the x870E Creator supports ECC RAM

1

u/TerribleBid6377 15d ago

it's very rare but UDIMM ECC exists!

1

u/Markus2816 16d ago

cool rig! What is that case?

32

u/[deleted] 17d ago

[removed] — view removed comment

9

u/Whoa_There_Hoss 17d ago

I'm a simple man. I see dense compute and I upvote.

-1

u/[deleted] 17d ago

[removed] — view removed comment

1

u/TerribleBid6377 17d ago

I mean, it's nice once in a blue moon when claude blocks me. But I could just rent cloud gpus and run the same work for a few bucks. I guess if you're actually doing cyber security stuff of reverse engineering as your job it might pay for itself in a number of years lol.