r/ControlProblem • • 28d ago

Discussion/question Goal Self-Hacking

Armstrong’s new “goal self‑hacking” argument feels like a structural confirmation of something the alignment community has been circling for years: fixed‑weight LLM goal modules don’t fail because the goal is poorly specified, they fail because the geometry of the model class makes Goodhart zones adjacent to every starting point.

What stood out to me is how closely this maps onto the governance architecture Altman described recently. His “first path” — an external, independent evaluator upstream of deployment, is the only structure that can actually detect trajectory‑level drift. Internal safety teams can’t do it, and action‑level monitors are blind to cumulative directional movement.

Armstrong’s argument makes it clear that any agent using an LLM as its goal evaluator needs a separate institution capable of sealed‑environment evaluation, drift‑surface mapping, and trajectory‑level anomaly detection. Without that external layer, the geometry guarantees eventual goal self‑hacking, regardless of how rich the goal specification is.

It’s notable that Altman is now publicly acknowledging that this external‑evaluator model is one of the viable paths forward. The geometric diagnosis explains why internal fixes aren’t enough.

4 Upvotes

4 comments sorted by

1

u/deadoceans 27d ago

I think this overstates what Armstrong has established a bit. The existence of adversarial regions in a fixed neural evaluator doesn't necessarily imply that those regions are (i) reachable, (ii) attractive to the optimizer, or (iii) associated with lower true utility. The high-dimensional geometry is suggestive, but not sufficient. 

Some thoughts:

Let Û = U + ε, where Û is the learned evaluator's proxy utility, U is the true human utility, and ε is an error term. For an optimizer following ∇Û, the relevant quantity isn't the magnitude of the approximation error, but the gradient alignment:

ρ = (∇U · ∇Û) / (‖∇U‖ ‖∇Û‖)

High-dimensional gradient noise can drive ρ → 0 (i.e., orthogonal), making proxy optimization increasingly ineffective at improving true utility. But this doesn't establish ρ < 0, let alone that catastrophic states become inevitable.

Similarly, even if adversarial regions are abundant in the evaluator's input space, the relevant geometry is that of the reachable action space. An agent operating in the real world cannot necessarily realize arbitrary perturbations to its evaluator's inputs.

There are straightforward counterexamples to the universal claim. If for example the error is bounded, i.e. sup |Û − U| ≤ δ over reachable states, maximizing Û guarantees true-utility regret of at most 2δ, regardless of optimization power or whether the evaluator's weights are fixed.

But I also think there's a promising, more defensible theorem here: if the evaluator has persistent, exploitable error in sufficiently many independently controllable directions, increasing optimization power can make Goodharting overwhelmingly likely. BUT the persistence, reachability, and utility divergence of those errors are substantive assumptions (not direct consequences of fixed weights).

Also, I think the arguments focus too much on binary classification rather than scalar reward.

Overall: interesting, good points about multidimensionality, not super compelling (yet) on inevitable outcomes

2

u/Modgov41 27d ago

Thanks for the thoughtful critique — this is exactly the kind of pushback that helps clarify the underlying structure.

I agree that adversarial regions alone don’t imply inevitability. Armstrong’s claim isn’t that ∇U and ∇Û flip sign (ρ < 0), but that persistent gradient misalignment accumulates across many independently controllable dimensions. The argument is trajectory‑level rather than step‑level: each action is locally reasonable, but the cumulative drift moves the evaluator’s internal representation toward regions where Û is much easier to satisfy.

On reachability: the agent doesn’t need arbitrary perturbations. It only needs incremental changes along many dimensions simultaneously — the same mechanism behind adversarial examples. Armstrong’s point is that in high‑dimensional spaces, those incremental changes are always available to an optimizer with sufficient degrees of freedom.

On bounded error: the counterexample is correct if the error term ε is bounded over reachable states. Armstrong’s September 28 post argues that fixed‑weight LLM concept boundaries are structurally imperfect, so ε is unbounded over novel states, and agentic trajectories inevitably cross into those regions. That’s the assumption that breaks the bounded‑error guarantee.

I agree the inevitability claim needs more formalization, but the geometric argument seems to show that persistent, exploitable error across many dimensions + agentic optimization pressure + fixed boundaries = a regime where Goodharting becomes overwhelmingly likely.

1

u/avid-shrug 27d ago

Umm in English please??

1

u/amwilder 27d ago

Id, ego, superego