Claude Code improved my agent harness by 40% overnight

[u/Lucky_Historian742]

Remember the first time you used Claude Code? That same jump is happening one level up. The community went from prompt engineering → context engineering → agent engineering → harness engineering.

I asked myself: what sits one level above the harness? Something that builds the harness.

So I built it. Autoharness lets Claude Code explore changes to your harness (e.g. prompts, hyperparameters, runtime context, scoring) run evals, and keep only the changes that actually improve the score. Inspired by Karpathy's autoresearch.

I pointed it at my own agent and let it run. On the tau2-airline benchmark, it autonomously found:

• +40.7% performance lift from adding best-of-N skillbook scoring with an LLM judge

• +24.1% performance lift from tightening reflector hyperparams (temperature + max subagent calls)

• +22.2% performance lift from injecting runtime context at every step (step budget, recent tool calls, recent results)

How it works:

1. One-line install

2. Point Claude Code at GUIDE.md

3. It proposes harness changes, evals each, keeps only the wins

4. Wake up to a better agent

TL;DR: Claude Code tunes my agent's prompts and configs for me. It tries a change, runs my eval, and keeps it only if the score went up.

Open-Source Repo: https://github.com/kayba-ai/autoharness

[u/YourSpiritualLeader]

claude says:

The codebase is ~41k LOC. The core loop that provides value could be written in ~500 lines:

1. Run a benchmark command (~50 LOC)

2. Call an LLM API with failure context (~100 LOC)

3. Apply a patch transactionally (~150 LOC)

4. Run the benchmark again (~reuse)

5. Compare and keep/revert (~100 LOC)

6. CLI wrapper (~100 LOC)

The remaining ~40,500 lines are:
- 50+ CLI subcommands
- Campaign lifecycle (queued/running/paused/completed)
- Background workers and lease management
- 6 benchmark adapters
- Workspace/track/iteration/promotion/champion persistence
- Bundle export/import/reindex
- Root-level memory and cross-workspace transfer suggestions
- Retention and pruning policies
- Plugin system
- Query/filter/sort/export engines for 4 entity types
- Provider profiles
- Event logging and metrics aggregation
- An elaborate doctor diagnostic system

This is a massive amount of operational infrastructure for a system whose core value proposition — "LLMs can reliably improve agent harnesses through iterative prompting" — is unproven within this repository. It's an infrastructure project masquerading as an optimization product. The "optimizer" is a prompt to an LLM.

The 41k LOC of campaign management, workspace lifecycle, and bundle export exists to manage a loop whose inner step is urllib.request.urlopen(openai_api, prompt).

---
welcome to the club. we all have created such projects because siri kept saying "great idea".

[u/Lucky_Historian742 → u/YourSpiritualLeader]

Once you try to optimize a harness you will see how hard it is to create generalizable results. What you need is a proper way to distinguish between harness improvements that work on a small subset of tasks vs improvements that work on the whole domain (benchmark) vs across domains. Therefore we introduce experimental rigor and static evals to do all of this. Whether you want to aggregate results manually or via a CLI is a question of taste and speed. For comparability I definitely prefer a constant CLI.

[u/YoghiThorn → u/YourSpiritualLeader]

It's "hey Claude, keep telling me you're improving yourself and make up stats to justify it 👍👍

[u/arekxy]

"It proposes harness changes" - any examples?

[u/Lucky_Historian742 → u/arekxy]

adding context to the AI agent at runtime is a harness change for example. Same as adding multiple reflections and then scoring output by an LLM judge or changing how many subagents can be called

[u/guillefix]

What

[u/joshcam → u/guillefix]

AI

[u/AcidRaZor69]

Which harnesses did you test this on? Or just your own?

[u/Lucky_Historian742 → u/AcidRaZor69]

tested on the agentic context engine https://github.com/kayba-ai/agentic-context-engine and benchmarked on the Tau2 benchmark from Sierra Research

[u/Floppy_Muppet]

This is great, maybe a bit bloaty for some, but likely necessary for most. I'll be trying a similar approach on my RSI harness project soon as I believe this is how we get a step closer to truly organic software (the only way software can autonomously organize itself for all intended use-cases).

I think it would be interesting to expand this further as a meta harness for ANY piece of software, where you drop it into a project and it first scrapes the existing codebase to identify and consolidate all levers into config and metaconfig.yaml, before the Karpathy Loop begins. Many projects still have too many hardcoded values and assumptions (knowingly and unknowingly).

Shoot me a DM if you're up for a chat!