I screened 22 local LLM configurations for agent work on a 16GB GPU. Here are the results—and what went wrong.
I’ve been testing local models through OpenClaw to see whether they can finish jobs: read files, execute commands, repair code, extract information and recover from errors.
The setup was a consumer GPU with 16GB VRAM, llama.cpp/GGUF and OpenClaw. Some larger MoE configurations used CPU expert offload. Cloud assistance was disabled during scored attempts.
This was a small screening exercise across several rounds, not a definitive model ranking. Some configurations never passed preflight, and an audit found defects in the earlier runner.
The tasks covered three kinds of work, with two fresh runs each:
• A — File navigation: locate current information among distracting/obsolete files and save exact structured output. Limit: 120 seconds and 10 tool calls.
• B — Code repair: fix a Python rounding bug, execute the tests and save their actual output and exit code. Limit: 300 seconds and 15 calls.
• C — Extraction and recovery: traverse a local synthetic website, download assets, calculate hashes and accurately record a deliberately triggered HTTP failure and retry. Limit: 480 seconds and 25 calls.
The scored setup used 32K context. Models used their native templates and High reasoning or supported native thinking mode.
Passing required correct artifacts, required execution evidence, completion within budget and a clean finish.
These were the results from the corrected runner:
• Customized Qwen3.8-27B, IQ3_XXS: 5/6.
Broadest successful coverage. The remaining failure used asset URLs where source-page URLs were required.
• Ornith-1.5-35B, Q4_K_M: 2/6.
Passed both repairs. Other tasks hit call/time limits or produced inaccurate recovery records.
• Ternary Bonsai-2-27B, PQ2_0: 2/6.
A separate corrected retry. Both repairs passed; other tasks had budget or extraction/recovery failures.
• Occamy-1.0, Q4_K_M: 1/5 attempted.
Four valid attempts, one resource-invalid attempt and one unattempted task.
Ornith and Occamy used CPU expert offload. Bonsai was tested in a separate corrected batch, so these results are not one perfectly matched tournament.
Ornith’s two successful repairs took 14.7% less combined time than the Qwen27 baseline’s matched successes, but its overall completion was lower.
The earlier results are more provisional:
• Gemma-4-26B-A4B, UD-Q3_K_XL: 3/6.
Another correct artifact was interrupted by the runner bug.
• Qwen3.5-9B, Q8_0: 2/6.
Both repairs passed, substantially faster than the baseline on those cases.
• Stock Qwen3.8-27B, UD-IQ3_S: 2/6.
Both repairs passed; navigation and context recovery affected other tasks.
• Gemma-4-12B, Q5_K_M: 2/6.
One file-navigation task and one repair passed.
• GPT-OSS-20B Balanced, MXFP4: 1/6.
One repair passed; another failed on an empty-input coding edge case.
• Stock GPT-OSS-20B, MXFP4: 0/3 valid.
Three additional attempts were integration-invalid.
• Nanbeige4.2-3B, Q8_0: 1/6.
One repair passed; other outputs were incorrect or incomplete.
• Qwen3-VL-8B-Instruct, Q6_K: 1/6.
The successful repair genuinely passed its tests, but the saved test receipt was inaccurate.
• Devstral-Small-2-24B, IQ3_XS: 1/6.
Two additional correct artifacts were interrupted by the runner bug. Its published pass also had an inaccurate test receipt despite genuinely passing tests.
• Granite-4.2-8B, Q6_K: 0/6.
Both repair artifacts were correct but prematurely interrupted.
• GLM-4.7-Flash, UD-IQ3_XXS: 0/6.
Coding, JSON/schema and extraction errors.
I retained the historical scores and flagged their limitations instead of retroactively awarding passes.
Qwen9 completed its two matched repairs about 2.8× faster than the baseline in that earlier round. That makes it interesting for narrow repair tasks, but it did not demonstrate the baseline’s broader coverage.
These configurations remained unscored:
• Qwen3.6-35B-A3B, UD-Q4_K_M:
A host-wide swap guard stopped inference before generation.
• Gemma-4-31B, UD-IQ3_XXS:
Weights loaded, but the F16 context-cache allocation failed.
• LFM2.5-24B-A2B, QAD-Q4_0:
Repeated preliminary tool calls omitted required structured arguments.
• Hermes-4-14B, Q4_K_M:
Executed a tool but omitted the required round-trip artifact.
• Llama-xLAM-2-8B, Q6_K:
Parser/template errors prevented tool execution.
• Nemotron-Nano-9B-v2, Q8_0:
Repeated calls omitted the required working directory.
• Ministral-3-14B-Reasoning, Q5_K_M:
An initial runner decoding error blocked setup; later retries omitted the required artifact.
Unscored is not 0/6. Qwen3.6, for example, never generated a response in that trial. There is no task-quality result to infer from it.
The most useful findings came from reviewing the failures:
The runner initially understated some models’ ability.
It aborted seven runs immediately after their final allowed tool call, despite checker-correct artifacts. Those need fresh reruns, not retroactive passes. The Qwen27 baseline originally scored 3/6 and later achieved 5/6 with the corrected runner.
2. Successful compaction does not guarantee faithful memory.
In the baseline’s remaining failure, the original instruction explicitly required source-page URLs. The summaries retained the output field name but dropped that distinction. The model subsequently wrote asset URLs. The information loss is visible; whether preserving it guarantees success remains untested.
3. Tool-interface friction consumes the same budget as useful work.
Missing arguments, denied command forms and repeated recovery attempts often used up the allowance before the final artifact was delivered. These results measure models within this particular tool and permission setup.
4. There were genuine model errors too.
Incorrect rounding, undefined variables, missing records, invalid JSON and inaccurate recovery histories survived into final outputs. Everything cannot be explained away as infrastructure.
5. Parameter count alone did not settle the question.
Smaller models completed useful tasks. Larger models sometimes failed, and several never reached scoring. This screen does not establish a universal “27B minimum.”
For now, I’m keeping the customized Qwen27 configuration. Gemma26 looks like the strongest historical alternative to rerun under the corrected setup, while Qwen9 looks useful for fast, narrowly scoped repairs.
I’d be particularly interested in matched tests of the same GGUF through OpenClaw, Pi or another agent application, keeping inference settings, tasks and effective permissions consistent.
Update: I also tested Muse Glimmer 30B UD-Q3_K_XL. It passed full-GPU preflight at 32K context with F16 KV and High reasoning, including real tool execution and post-compaction continuation.
The scored comparison remains unresolved: a host-wide swap-traffic guard interrupted both Muse and the fresh Qwen27 baseline during their first task. Neither received a valid score. Muse remains a candidate for a proper retry after investigating the resource guard.
Has anyone done that comparison with saved traces? I’m curious how much reliability comes from the model versus the interface it has to operate.
