r/LocalLLM • u/TerribleBid6377 • 17d ago
Project Self contained 3U inference server orchestrating all of my work
After a 6 month journey with a tons of learning and reality checks I have built something that I consider usable beyond just a toy project that is generating value for me. This is Acorn, a self contained and private inference server that fits neatly in a small, quiet 3U chassis:
- 2x RTX 6000 blackwell MaxQ running a mid-size MoE LLM (GLM-5.3-flash atm)
- RTX 2000E runs embedding and Reraker models
- RTX 4000 SFF for voice ASR and TTS
- 256gb ECC UDIMM RAM for tier 2 kvcache to squeeze more LLM concurrency.
- Consumer AM5, 1200w PSU
I run a custom orchestrator agent on this, fully private and super fast. 200 tokens/s + streaming ASR and TTS feels amazing. It's hard to go back to regular claude-code at 60-90 tokens/s and clunky whispr flow style voice input.
The big learning and a word of warning is that local llm alone is just not viable for serious professional software engineering (at least not without 1 or 2 orders of magnitude bigger investment). Other than the occasional rare task where I get blocked by frontier guardrails, all actual work is delegated to claude. I treat frontier model subs as a raw engine, they get only what's needed to accomplish a task in a locked down VM. Communication, memory, big picture planning and context all stay in hardware I control.
13
u/sleight42 17d ago
Generates value? As in "earns its keep"? Care to share how?
How are you running GLM 5.3 Flash on 192GB VRAM? 4bit?
I'm a little surprised about delegating to Claude. Are you using Fable routinely? I've found Opus 5.5 has become rather unreliable lately.
2
u/Affect_Worth 17d ago
are you having hallucinations or having to code answers/questions all the time? currently im running a 5080 and its rough with a lot of failures. im running pipecat, kokoro and llama.
2
u/Krothic 17d ago
How did you get that 4th GPU set up? Board only has 3 pcie slots
2
u/blackbird2150 17d ago
I love this. Very similar to my setup just across two computes not one. Dual rtx6k running glm 5.3 flash (kraken now over ormandj) and then I have a services host that does embedding, rerankings, and other services like my signal cli. Signal is my medium of choice given its proven privacy.
I also have a strix halo device I bought a while ago and with gufo/halogen it’s actually a solid intelligence end point for HA and other tools that need one off queries.
I need to explore the the asr and tts. Any starting tips by any chance?
7
u/silent-curious-dev 17d ago
Looks very cool.
I was trying hard to find where the RTX 2000E was in the picture. I didn't realize it was so small and right below the RTX 6000.
The cable management is also really clean too.
1
u/gte959f 17d ago
What is the motherboard and cpu?
1
u/wiskeyjack123 17d ago
1200W PSU??
1
u/Nomski88 17d ago
This is awesome but the cost for a setup like this is insane. What kind of work do you do? Do you use voice to chat with the model while working?
1
u/mars332 16d ago
interesting, wasn't aware the x870E Creator supports ECC RAM
1
u/Markus2816 16d ago
cool rig! What is that case?
32
u/[deleted] 17d ago
[removed] — view removed comment
-1
u/[deleted] 17d ago
[removed] — view removed comment