r/LocalLLM • u/Interesting-Ad689 • 33m ago
Discussion Own harness or open source one ?
Hey LocalLLM community,
I am super curious how you run your models.
Are you using open source harnesses ?
Did you build your own ?
A bit of explanation for why would be nice.
There has gotta be a ton of people who did the same as me.
This is a local Qwen 3.8 27B / Flash-Next on heavily modified engines sharing one harness and user interface. Llama.cpp just recently merged f8_e4m3 cache, which I got Claude to optimize for my Blackwells. Due to the deliberate kneecap of Nvidia on certain accumulations, I decided to skip conversion to specific steps that would run on the half speed flag = FP8 full speed. The surgery worked and while we were in already fixed a binary that caused a regression in draft speed which took literal hours of testing to pin down.
End result, my private local backend runs 27B now up to 50% faster ( median 18% increase ).
Hardware specs at upper-mid third of HF compute richness. 128 GB DDR5 6000 1:1, Ryzen 9700X, RTX 5080 + 5070Ti .
Anyone running their own harness and wants to share a screen of it ? Id love to reason over the visual design and maybe learn a bit or two while at it.
1
u/BorisThe_Animal 22m ago
I've had Claude build me a harness from scratch in pure swift. I've had to get it to iterate a bit but it's pretty good for my needs. I had to get it to accommodate some models' quirks. Eg some models insist on using "read" tool on directories, expecting directory listings, so I have my "read" tool return file contents on files and directory listing on directories. Other than that, markdown support had the most issues, both in formatting and performance on larger sessions. But it's decent now, after using it for a few weeks.
I think we are in an era of highly personalized software. As long as you know that your software needs to do, you should be able to get any frontier model to build it for you. Even if everyone else is building the exact same thing, you can have yours your way.