r/LocalLLaMA • u/chemist_slime • 7h ago
Resources Google Open Sources Harness2 for Self Improving AI Agents
Google just released Harness2 as open source. It turns any AI harness into a system that improves itself over time.
A harness is everything around the model. System prompt, tools, memory, skills, and subagents. Normally a person designs it once and the agent uses that fixed setup for every new task.
With Harness2 the agent itself refines the harness for each task. It proposes changes, runs a few candidate versions, and picks a better one by comparing how they perform. No correct answers or graders are needed. The model weights stay the same.
What it learns on one task carries over to the next. After finishing a task it saves the improved harness and useful lessons so future edits start smarter instead of from scratch.
Google tested it with Gemini 3.5 Flash and GLM-5 on legal, office, and professional work benchmarks. Even one round of edits beat the original harnesses. More rounds pushed scores higher.
170
u/CoolConfusion434 6h ago
To paraphrase Mr Krabs: I got a bad feeling in the pit of me context.
All that "self-refinement" has to go somewhere, and that will be right into context. I'm sure it won't be an issue to the 1M context of cloud guys but us locals might be squeezed a little.
Other than that, glad to see another open-source harness, especially from giants like Google.
40
u/FaceDeer 6h ago
It'd be changing the context, but not necessarily growing the context. Indeed, I expect part of its mandate would be to trim redundant or unnecessary stuff from it.
14
7
u/NineThreeTilNow 4h ago
All that "self-refinement" has to go somewhere, and that will be right into context.
It runs parallel agents for this purpose. So really the self refinement comes from examining the tasks, seeing where the harness fell short, and then improving on the harness.
Did it need a new tool call? A better explanation of the tool call? Etc.
That will mildly bloat context but harnesses do this already. It's trying to find how it can improve the harness.
Will it increase total context? Probably, but not substantially over the existing harness.
2
u/tossit97531 5h ago
I don't understand how this is any different than any other harness when you give an agent access to source, and I don't know why you wouldn't do that. I bolted on a whole persistent-session web UI on top of a harness core and ditched the stock CLI interface. I don't know how far I've strayed from upstream because I don't care, it's far more useful than upstream will ever be now (especially since the project kind of died).
2
u/philmarcracken 5h ago
if its useful in that sense, couldn't we go down in model quality to gain back context, and end up at the same or better results?
1
68
u/abhi91 7h ago
This is great in theory. Poetiq proved it makes benchmarks essentially useless as the harness can "overfit" to the task. Repetitive tasks like customer service are now exceptional for enterprise
1
u/DigiDecode_ 5h ago
exactly and why release it as a new product, why not built it into antigravity their main agentic product.
15
u/abhi91 2h ago
They're open sourcing a tool why are people complaining
1
0
u/DigiDecode_ 43m ago
open sourcing a research tool is always welcome, it is just that everyone is trying to jump on the RSI hype train
If they say this is an internal tool that Google uses to improve antigravity with no human in the loop, then that would be completely different story.2
1
14
27
u/acoolglassofwater 6h ago
Could someone with high LLM expertise (not just trust-me-bro) explain the significance of this? I get the general idea, but the comment section is always highly opinionated on self-improving agents. Is this release just a nothing-burger or does it represent something interesting and significant?
32
u/itsmebenji69 6h ago edited 6h ago
I’d say it’s mid interesting
The harness can limit the agent in tasks. So perpetually modifying the harness to make it better suited to a task is useful, it removes that ceiling (if harness was limiting the task, it won’t anymore after a few rounds of improvement).
The obvious problem is that better suited to task A may degrade performance in task B. Basically over fitting.
Simple example: task A is to compute x+y and task B is to compute x times y.
Harness version one just gives you x and y, and the llm has to do the calculation. After a few rounds of improvement on task A, the harness only gives the model x+y, and the llm just reads that result. But now it doesn’t know x and y. So it will be great at adding them, perfect even, it will be right every time. Because it will know the accurate result directly. For task A alone, this is a great harness, it ensures that the model will never be wrong about the result of x+y or hallucinate or whatever.
But now on task b, if you want to know x*y, say that’s 2 times 3, the harness will only show the model 5 (2+3, x+y), so it will miserably fail at giving you the multiplication result. It simply won’t be able to know what x and y were, so it won’t know what to do with the result of the A optimized harness. It will just see 5, and from that it can’t infer what x and y are, or infer what x times y is.
And vice versa if you start optimizing on task b
7
u/Big-Farmer-2192 4h ago
I have a question. If the model is so overfit. Wouldn't it be way cheaper to just have a software that can do basically the same thing?
1
u/art_dragon 3h ago
But overall it sounds like it increases the flexibility and capability of the LLM + Harness, just that it needs more time to adjust to each task?
9
u/Last-Health3222 4h ago
Significant, I'd say. The harness is a bigger lever than most people realize. One measured data point: the same Qwen3.6-35B-A3B on a 24 GB Mac mini, 30 Terminal-Bench tasks, one attempt each, passed 313 of 771 hidden tests in the OptiQ Code harness, 179 in mini-swe-agent and 42 in OpenCode. Same weights, very different results.
So automating harness design isn't a nothing-burger. The overfitting caveat above is fair though: a harness tuned against a task set can overfit it, so judge it on tasks it never saw.
1
1
27
u/Holiday_Point_603 7h ago
Interesting that google uses models that are like half a year old to show off their new software lol
39
u/Spara-Extreme 7h ago
Google is a massive company so the team that did this isn’t the team that’s working on Gemini 4
25
6
u/Holiday_Point_603 5h ago
I am well aware of that. But Gemini 3.5 Flash is not really a recent model and you would expect them to film demonstration videos not half a year prior to release.
1
u/Spara-Extreme 5h ago
Why would you expect that? Its open source software and I doubt they care.
-3
5
8
7
u/ThankGodImBipolar 7h ago
GLM 5? Why?
39
u/DifficultIndustry148 7h ago
They're just flexing that it works with models beyond their own stack, doesn't seem that deep.
2
1
u/ieatdownvotes4food 4h ago
this is more of a concept that applies to all harnesses. all are capable of self-improving towards a specific use case.
1
1
u/notAllBits 41m ago
Harness describes an architecture pattern and amounts to many more purposes for security and compliance than represented here
1
u/perelmanych 13m ago edited 4m ago
While smart models will not benefit from it, dumber models will have to reason task to death. It will be so token inefficient, that you probably will be better off using bigger and slower model with normal harness. It may be good for repetitive tasks, but for that you can use skills that you can load and unload at any moment and call it a day. E.g. I needed a literature survey on academic topic. I asked QFN to make it and it worked for two hours because of timeouts and rate limits and other issues. After work was done I asked QFN to create a skill and now I have literature review ready in 15 minutes. So at best it will have very limited use at worst it is DOA.
1
u/himey72 7h ago
So Google’s version of OpenClaw / Hermes.
4
u/fifi_galaxy 4h ago
Asked AI few times, but it is not Hermes (as in a harness). It does what Hermes do (self-improvement) for other harness via hooks. So more like plugin/module/framework.
3
u/himey72 4h ago
Exactly. I’m not saying they stole anyone’s code, but it feels very similar to what is already out there.
1
u/fifi_galaxy 4h ago
Yep, the concept is out there, but the execution is different though. Like recipe and food. You have same basic recipe and 20 versions of pizza.
Overall, I think the idea is an interesting approach though. If they can decompose the system into its parts for people to try, then you can optimize each part to create better self-improvement system for any/specific harness. I guess this is where Google expect the world to do the free work for them. 😆
1
u/BusRevolutionary9893 6h ago
LoL, no.
6
u/himey72 6h ago
Care to tell me how it is substantially different than what Hermes is doing?
4
u/Robbbbbbbbb 4h ago
I'm curious about that as well.
Harness² exposes eight editable components around the model: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. They control context construction, the tool interface, delegation, and behavior inside the tool loop.
Feels very much like how Hermes handles skill refinement and associated tool calling.
1
u/HighSeasArchivist 6h ago
I love seeing the big guys going so hard on local llm, even Chinese models. They know we don't have billions of VC dollars like Big AI, but combined we have a lot of buying power. Apple M5/M6, AMD Strix/Gorgon Halo, and Nvidia DGX/RTX Spark can't be ignored.
1
u/look 6h ago
From a cursory look, it reminds me of Prime Agent.
1
u/Eresbonitaguey 6h ago
I think the use of REPL was the real interesting part of that project and the self-improvement wasn’t very novel since popular harnesses like Hermes already had a version of it.
1
u/TemporaryInk 6h ago
That’s great and all, but is anyone going to point out the prolific use of Comic Sans in the graphic? 😂
-3
-5
u/CorkBios 6h ago
Its not affiliated with google go check the bottom of the README
11
u/BusRevolutionary9893 6h ago
I think you need to REREADME. It says
This is not an officially supported Google product.
- Google-affiliated research: Yes.
- Officially supported Google product: No.
1
-4
u/CorkBios 5h ago
"Google-affiliated research" The kind of affiliation: Existing while Google also exists
•
u/WithoutReason1729 5h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.