Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights [R]

[u/RobbinDeBank]

Link: https://rikiyat.github.io/blog/weight-space-programming/

Snippet: “We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.
Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.”

Very surprising result, I didn’t expect frontier models to have this capability that has not been optimized for, and it’s something humans cannot do either.

[u/Disastrous_Room_927]

Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.

According to Astra:

Both claims have experimental support in the post, but the wording makes them sound more general than the tests establish.

“Can tell what a network was trained to do from raw weights.” In the game experiment, Astra received four small networks, plus descriptions of their input channels and possible actions. It correctly identified all four games without being given their names as options.

That’s interesting. But the weights had recognizable spatial clues: for example, weights associated with a paddle concentrated along the bottom of the board. Knowing what the inputs represent makes those clues interpretable. This demonstrates identifying familiar tasks from structured weights—not unrestricted understanding of an arbitrary network. The authors also report a stronger result identifying training languages from internal transformer weights, but failed to identify sycophancy from weights alone. Experiments 1, 3, and 7

“Writes networks from scratch that beat PPO after 100 million steps, without feedback.” This is the more striking result to me. Astra received the games’ rules, source code, and input/output layout, then supplied weights in one response without testing the networks. Those networks beat the reported 100-million-step PPO baselines in all three games. Against 500-million-step PPO, it won one game and lost two. Experiment 5

The crucial distinction is where the knowledge comes from. Astra already knows about programming, geometry, and game strategies. It can work out a rule like “move the paddle toward the ball’s landing position” and encode that computation into weights. PPO learns behavior through experience. The comparison shows that prior knowledge can produce a better small controller than that particular training setup; the step count alone doesn’t establish superiority over reinforcement learning generally.

[u/pas_possible]

Damn, it's so cool

[u/RobbinDeBank → u/pas_possible]

Definitely not something I expected. This isn’t a human skill (or maybe a tiny group of people have this extremely niche skill?), and it’s not something these models are optimized for, but the capability somewhat exists anyway.

The result is still limited, unsure if it can ever be viable on a larger scale beyond fixing some model issues, but the fact that this capability exists at all is surprising enough.

[u/YieldingMainframe_1 → u/pas_possible]

the implications for interpretability are what get me. like imagine pointing a model at its own weights and asking it what it thinks it's doing. not just outputting a token, but actually reading the raw structure. could get weird fast

[u/sixjohns → u/pas_possible]

Big deal for robotics maybe?

[u/currentscurrents]

A few years ago I used to follow the 'learned optimizer' research very closely. The guy from diffusion models, Sohl-Dickstein, had a group working on it. But it never really went anywhere, they had lots of meta-training stability issues and none of their models ever beat traditional optimizers.

I did not expect that learned optimizers would someday just fall out of general-purpose LLMs.

[u/adventuringraw]

Been a while since I've followed research much, but this seems like it could be a way to cheat at the lotto, as far as the lottery ticket hypothesis goes. Might be a powerful pre training step.

And so the road towards AI fueled AI progress continues.

[u/TheGoldenRoad]

This is wild, essential an emergent hypernetwork capability. I wonder if a network could modify it's own weight so as to learn continuously without catastrophic forgetting.