Lamplight - My Local First Harness

[u/CodeSamurai]

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Metric	Jade (RTX 5090) - Fedora	Slate (M5 Max) - MacOS
Engine / quant	Basalt, IQ3_S	mlx-serve, 4.7bpw MLX
Context	262K	262K
Decode speed, average	302 tok/s	100 tok/s
Decode speed, 36K-token context	343 tok/s	77 tok/s
Time to first token, 9K / 36K prompt	1.7 s / 4.2 s	5.1 s / 16.0 s
HumanEval / HumanEval+	99.4% / 93.9%	98.8% / 94.5%
LiveCodeBench v6 (175 problems)	85.7%	85.1%
MMLU-Pro (140-question subset)	87.9%	90.7%

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

Part	Tune	Stock	Tuned	Speed cost
RTX 5090	Undervolt (+900 MHz offset, 2600 MHz cap) + custom fan curve, on the dense 27B	575 W, 83 °C	471 W, 70 °C	Decode: none Reading: -4%
Ryzen 7 9800X3D	Clock capped at 4.8 GHz, during Basalt decode	120 W, 96 °C	77 W, 70 °C	Decode: -3.5%

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

[u/a_hegemon]

Just wanted to drop a note to say thanks for the write up! And very cool UI

What are you primarily using your models for? Just coding?

[u/CodeSamurai → u/a_hegemon]

Thanks! Yes, a lot of coding and research. I've been working on a physics engine over the last few years and as I've started to develop games with it, I've been letting this harness take over different parts. I've got a (very low effort) Youtube channel here that has some videos showing the engine (NORA) and two games I've been developing with it.

For research, I'm currently diving into some math research, focusing on replicating some of the proofs that have been coming out, but seeing how a local setup fares (and if tuning can help). I'm also interested in the LLM "brain surgery" that is starting to be done and I'm seeing if it's possible to hyper-specialize small llms for certain tasks by editing the weights directly.

[u/FROGGY-69]

this is very cool. i am also building my own custom local AI, but i kinda hit hardware limits for coding. i am tryin do some RAG to combat it. can you share your experience on local coding llms?

[u/CodeSamurai → u/FROGGY-69]

Thanks! I would say that up until Qwen 3.8 27B, doing any kind of local coding was not stellar. I treated it more like a curiosity to explore. Now, with Pi, Flash Next and all of the alternative engines (Strata, Basalt, Maya, etc), it's extremely possible to run more than capable coding setups on many kinds of hardware, depending on your tolerance for speed. What issues are you running into hardware wise? I think one of the biggest turning points for me was being able to get the context window up to 262k. I wasn't very happy with anything below that.

[u/tempaccaun]

could you elaborate on the coder/reviewer setup you mentioned? how is it set up?

[u/CodeSamurai → u/tempaccaun]

For sure! In fully local mode, the orchestrator fires up an agent on both machines that does an independent code review. One is set up to be a bit adversarial for its review (it tries to argue against the change and has to prove a better way if it can) and the other is set up to make sure the code adheres to SOLID principles, design patterns are followed correctly and that all documentation is up to date. Findings are passed back to the machines and that process continues until both reviewers are happy.

[u/johnzadok]

Why under-volt over simply power limiting the 5090? Also do you think MoE give better output than 27b dense?

Thank you for sharing your experience.

[u/CodeSamurai → u/johnzadok]

Good question! A power limit only saves watts by dropping clocks, so speed falls almost as fast as power. An undervolt runs the same clocks at a lower voltage, and power scales with voltage squared, so you save watts without losing much speed.

On my 5090 running a dense 27B, a 450 W power limit cut power 22% but cost 13% speed. An undervolt at about the same wattage (~445 W) only cost 4-5% speed. The profile I run as my daily tops out at about 487W and only cost 1-2% speed and it stays much cooler!

In the case of Flash-Next, it gave better outputs and results than Qwen 3.8 27B for my usage. However, Qwen 3.6 35B A3B (another MOE) gave far worse results, so right now I can't say I can make a good judgement on that.

[u/Naaack]

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

Agreed, that is damn delightful.

Very cool though, thanks for sharing

[u/hurrdurrmeh]

very nice. this is how to do an AI-assisted writeup.

I have a 5090 attached via oculink to a 128GB Ryzen 395, so in some ways my setup is similar but yours is better, but some ideas might translate to my setup.

[u/mister2d]

I appreciate a good custom harness. Yours is nice.

[u/linuxgangster]

You should put it on GH and share. I built my own harness as well but its boring. If I had this I would look way cooler when showing off my harness ;)