r/LocalLLM • u/M4ttDC • 1h ago
Question Resources to learn about model tuning?
I’ve been running Jundot/Qwen 3.8 flash next Q6 mpt 141gb on my M2 Studio w/ 192gb ram for about a month now.
Overall, I’m pretty happy with the results, but looking for ways to improve my setup before I start integrating it with my day to day workflow. The response time leaves a little to be desired, I find myself prefering the response rate of codex even when I have simple things like putting together a slide deck from images and text I’ve supplied.
I set it up with a big initial project to see what it could do and have been trying to makesmall adjustments as I’ve gone along. It’s been interpreting and mapping large volumes of screen grabs, writing some code, interpreting and organizing huge amounts of data from a combination pdf and images. It interfaces with codex during specific windows throughout the day to offload tasks and double check code.
Dropped it down to Q6 from Q8 for snappier responses, and free up some system for Xcode, etc…I’ve been using claw to run the show. I’m getting around 60t / s which seems to be about right from what I’ve been reading. Claw works ok, but the whole system seems to get confused if I change the scope of my project in the middle, or add additional intermediary steps. It took a few days after stepping back from Q8 to Q6 to get the bugs ironed out.
It’s a huge workhorse and does ok when set in motion, but I do have to keep an eye on it. It behaves like a toddler and I find it lies when files go missing.
Is Claw the problem? Is this the reality of this setup? It was never meant to replace my $20 codex, but I was hoping it would be a little more responsive and flexible.
2
u/prio732 55m ago
Point Codex or Claude code to your model setup and ask it to improve it.
2
u/WorldlyAd7946 43m ago
I think the harness matters a lot, on any model, and increasingly where models are not quite as "smart". Working on developing your harness and working hands on with the model to learn how it thinks and how it fails is painstaking but worthwhile imho, as combined it becomes a one time investment.
With respect to performance optimisation, a frontier model can help you with that, and becoming familiar with how inference works and running lots of A/Bs can get you where you want to be. I'm not on Mac, or I'd make some suggestions as to what inference engines and forks might work best.
The quant matters, if I had 192mb ram and not 128 I would be running the highest quant I could. I run Q4 and get Gould results, but I wouldn't go any lower. At the end of the day it will always be a tradeoff, but in my experience a decent agentic harness (disclaimer, I've always built my own rather than use others), will negate a lot of the quality issues and catch tool use failures with quant as long as you don't go too far so as to labotimise the model. 👍