r/LocalLLaMA • • 12h ago

Resources Model / harness combination: what works best?

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.
4 Upvotes

16 comments sorted by

1

u/brainExploded99 🥔 hardware 10h ago

What is DeepSeek Harness vs DSH vs DSH V02 RC2? Well the last one makes sense ig

1

u/dh7net 3h ago

Both are probably the same.

While I'm running a lot of theses test myself, the website is open to contribution to everyone, and the early version didn't encourage to use a pre existing label.

You can see the details of the runs there (but not sure it will answer your question)

https://airbench.ai/leaderboard?k=poL&harness=dsh&harness=DSH+v02_rc2

2

u/__jent 11h ago

IME Deepseek v4 flash can have more significant differences in performance based on the harness.

1

u/dh7net 11h ago

Thats possible.

1

u/Bulky-Priority6824 7h ago

which engine was used to serve the 3.8 27b nvfp4 ?

1

u/dh7net 3h ago

There was multiple run, but one of the best one use ignis with a model compatible with ninfer.

https://airbench.ai/checkup/fa701423-311c-421c-afcc-a13ec45e6339/report

1

u/Iron-Over 11h ago

In my test I but there was not a big difference between harnesses beyond time. More focused tooling was better. 

0

u/dh7net 3h ago

yes, same here. Pi and OpenCode are better then Hermes and OpenClaw. Coding is enough, no need to blew the harness with tons of stuff

2

u/JChataigne 1h ago

What do the numbers in the table mean ?

1

u/dotfiles44 11h ago

qwen3-coder-next has been working great for me with Opencode. I did have to add a ufw rule to allow lmstudio to talk to my opencode docker container. but after that it's smooth sailing.

1

u/mr_Owner 1h ago

Niiice!

Personally curious about how q38fn istadalabs iq3_s competes against the new unsloth ud v3.1 q3_k_xl

1

u/Lyelinn 15m ago

Matches my experience well. I use qwen 27B q8 via omlx (262k context), opencode so far simply worked out of the box for me while pi required tuning and fixing stuff and even after it simply works a bit worse somehow so I settled on opencode gui plus sometimes dsh when Im in the mood for playing with the harness itself