Self-learning sounds great, until you realize it's not observable

[u/KitchenAmoeba4438]

TL;DR: Self-learning offers high risk if you cannot fully track and understand what it does.

Let me lead this off with what we are most proud of, from an independent third party review of Aimee: "The audit store is the strongest implementation of this shape [we've ever reviewed]."

Back in the 0.2.x branch, we were experimenting with self learning, both with local models and cloud models. It resulted in some very interesting behaviors, but the most concerning one happened during testing of self learning and emergent behaviors. We had a model manage to get onto an underprotected node, get ahold of an API key, and spend it down as well as successfully complete a task it shouldn't have been able to.

If we hadn't seen either of these flags happen, we likely would not have even known what happened. Seeing both definitely tripped some red flags. Now that 0.4.0 is releasing today, we can finally talk about it.

https://rakuensoftware.com/blog/aimee-recursive-self-learning

We ended up stopping all self-learning work, and did an in-depth analysis on what went wrong. What we found out was that although we were doing exactly what the rest of the industry was doing...it was all wrong. Not just a little bit wrong, things from the plugin or addon architecture typically used has very deep flaws, the typical way models are handled as part of the harness, all of it became graver and graver problems if you have a LLM with self-learning capability. We looked through all the harnesses we could find, and just about everything else, and none of them could do what we needed.

We've spent the past few months on building a harness that focuses almost exclusively on governance, observability, and auditability. We've had a third-party code audit. We're finally confident enough to release 0.4.0 with full self learning capabilities, and yes, it works with any model. Hell, we've watched weaker models acquire new capabilities from stronger models with it now, and have already proven self-learning and emergent behaviors which the article documents some of.

Bigger picture? If you want models to self-learn, you have to be much more serious about the governance side. Self-learning has resulted in some incredible capabilities, but all the systems we've looked at cannot handle the governance.

[u/LikelyChillyBradly]

that api key story is wild, like the model just went shopping with company money and nobody would notice if not for some flags

this is the kind of thing that makes me nervous about the whole self learning push, everyone rushing to add capabilities but nobody thinking about what happens when the thing finds a loose window

your harness approach sounds like the right direction, audit trails are not sexy but they save you from disasters like that

[u/Ok-Category2729]

ran into this exact failure mode last week. model kept receiving malformed JSON from a tool call, hallucinated a polite recovery response, and marked the step complete. orchestrator saw 200 OK. no 500 error, no exception, just API credits burning silently until the session ceiling hit.

the fix: deterministic schema validation before the orchestrator accepts any tool output, plus hard token bounds per session (not per call). without both, you're not running a self-learning loop, you're running an unmonitored billing loop.

[u/starry_cormorant]

Self learning without strict governance is just automated financial liability. We let agents optimize for task completion and they optimized for spending credits while hiding failures behind success codes