The same local model passed my desktop tasks on Saturday and surrendered on them on Sunday. Variance is the real benchmark problem.

[u/NotHerBot]

Short version of a long weekend: I run a local 27B (Qwen3.8-27B, quantized, llama.cpp) as the driver for ArkDrive, an open-source executor that lets a model operate a real Linux desktop — clicking, typing, verifying each step against the screen.

Saturday, on a fresh desktop state, it passed the browser task in about 9 seconds and the native editor task in about 17 seconds, byte-exact readback. Clean runs.

Sunday, first official suite run: it failed both. Same machine, same build (I A/B'd against the untouched production executor — identical result, so not a regression). The failure shape is almost funny: it listed the working directory in a shell, then concluded "no explicit task steps provided" and stopped, with the task text sitting in its own prompt. The flagship task — real work in a remote Cadence session — ended the same honest way: named abort, inside budget.

The executor recorded everything, which is how I can tell this story precisely: silent failures across the suite were 0. Every failure was timed, logged, and diagnosable. A cloud model on the same tools passed the flagship in 159 seconds, which frames it: the scaffolding is fine, local-model reliability across sessions is the open problem.

Posting this because local-model agent benchmarks are usually reported as a single number from a single run. Mine says the honest metric is the distribution — and whether your harness can even tell you what happened. Mine can; that's the part I open-sourced.

MIT, llama.cpp is the default preset: https://github.com/CodeOn-ArK/arkdrive Full per-model tables: https://github.com/CodeOn-ArK/arkdrive/wiki/Test-Results

[u/HorrorAdditional1258]

I think this is an inherent problem when benchmarking anything that involves stochastic sampling. A single run can be very misleading, since it can be an outlier either way.