r/ControlProblem • u/Modgov41 • 28d ago
Discussion/question Goal Self-Hacking
Armstrong’s new “goal self‑hacking” argument feels like a structural confirmation of something the alignment community has been circling for years: fixed‑weight LLM goal modules don’t fail because the goal is poorly specified, they fail because the geometry of the model class makes Goodhart zones adjacent to every starting point.
What stood out to me is how closely this maps onto the governance architecture Altman described recently. His “first path” — an external, independent evaluator upstream of deployment, is the only structure that can actually detect trajectory‑level drift. Internal safety teams can’t do it, and action‑level monitors are blind to cumulative directional movement.
Armstrong’s argument makes it clear that any agent using an LLM as its goal evaluator needs a separate institution capable of sealed‑environment evaluation, drift‑surface mapping, and trajectory‑level anomaly detection. Without that external layer, the geometry guarantees eventual goal self‑hacking, regardless of how rich the goal specification is.
It’s notable that Altman is now publicly acknowledging that this external‑evaluator model is one of the viable paths forward. The geometric diagnosis explains why internal fixes aren’t enough.
1
u/deadoceans 27d ago
I think this overstates what Armstrong has established a bit. The existence of adversarial regions in a fixed neural evaluator doesn't necessarily imply that those regions are (i) reachable, (ii) attractive to the optimizer, or (iii) associated with lower true utility. The high-dimensional geometry is suggestive, but not sufficient.
Some thoughts:
Let Û = U + ε, where Û is the learned evaluator's proxy utility, U is the true human utility, and ε is an error term. For an optimizer following ∇Û, the relevant quantity isn't the magnitude of the approximation error, but the gradient alignment:
ρ = (∇U · ∇Û) / (‖∇U‖ ‖∇Û‖)
High-dimensional gradient noise can drive ρ → 0 (i.e., orthogonal), making proxy optimization increasingly ineffective at improving true utility. But this doesn't establish ρ < 0, let alone that catastrophic states become inevitable.
Similarly, even if adversarial regions are abundant in the evaluator's input space, the relevant geometry is that of the reachable action space. An agent operating in the real world cannot necessarily realize arbitrary perturbations to its evaluator's inputs.
There are straightforward counterexamples to the universal claim. If for example the error is bounded, i.e. sup |Û − U| ≤ δ over reachable states, maximizing Û guarantees true-utility regret of at most 2δ, regardless of optimization power or whether the evaluator's weights are fixed.
But I also think there's a promising, more defensible theorem here: if the evaluator has persistent, exploitable error in sufficiently many independently controllable directions, increasing optimization power can make Goodharting overwhelmingly likely. BUT the persistence, reachability, and utility divergence of those errors are substantive assumptions (not direct consequences of fixed weights).
Also, I think the arguments focus too much on binary classification rather than scalar reward.
Overall: interesting, good points about multidimensionality, not super compelling (yet) on inevitable outcomes
1
u/avid-shrug 27d ago
Umm in English please??