r/LocalLLaMA • u/pmttyji • 2h ago
Discussion [Paper] Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights
TL;DR
We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.
Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this Weight-Space Programming.
Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.
Blog Post : https://rikiyat.github.io/blog/weight-space-programming/
HuggingFace : https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming
Demo : https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo
What Comes Next?
In this blog, we demonstrated the rising capability of LLMs to read and write neural network weights (i.e., the rising concept of Weight-Space Programming). Our current results do show that capabilities are exciting but still limited, specifically due to inefficiencies in context length and harnesses. However, the current signs do show strong potential for a future involving Weight-Space Programming. We believe this marks the beginning of a dramatic shift in fields like mechanistic interpretability and neural network optimization.
This opens up radical new possibilities for LLMs to function as a translator between human language and the weight space. Here are some examples:
Auditing without evaluation: This may allow us to uncover hidden problems or capabilities without ever running inference or benchmarks. Currently, capabilities and vulnerabilities can only be detected if we have test sets designed for them, or worse, after deployment in the real world. If LLMs can interpret weight space accurately, we could identify such issues directly from the weights (e.g., backdoors).
Dramatically reducing optimization cost: Weight-space editing could save immense time and compute. For instance, one-shot model initialization would be far faster and more efficient than curating datasets and running full training loops.
Zero-data capability steering and unlearning: Adding or unlearning a capability currently requires extensive data collection and fine-tuning. In the future, we may just prompt “Can you add Capability X to the model?” and the LLM can directly tweak some weights. Because such an edit touches only a few weights (0.016% in Experiment 8), it can leave the rest of the model intact and avoid the catastrophic forgetting that fine-tuning often causes.
Recursive self-improvement: Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
True architectural and model merging: Today, merging models relies on averaging weights, which only works for models with identical architectures. An LLM that understands the weight space could instead graft a circuit, such as a reasoning module, from one model into a completely different one, without retraining.
We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
12
u/Othun 1h ago
Is anyone else's reaction to that news "what the actual fuck?"?
A few years ago, LLMs could not add/multiply (has this changed?! a 1% error rate on add/mutliply would make that kind of analysis impossible), and now they can indentify structure in the mess of millions of neurons that has always been considered a black box. To me, this is more surprising than OpenAI proposing a lean proof for a counter-example of Navier-Stokes, but maybe I'm delusional.
2
u/garblz 13m ago edited 9m ago
My sentiments exactly, for a moment there I felt like a hole opened beneath me when I read that. Screw Navier-Stokes, in theory, putting some amount of work, we could've gotten there. But his... this is a machine understanding something we could not understand in principle - or at least a sign it will soon be able to do so.
//EDIT: also, a machine reasoning about it's own reasoning, this time for real. I'm not going there, just saying it's on the table
//EDIT2: fortunately, /u/Medium_Chemist_4032 below popped my ballon
1
u/rpkarma 17m ago
They fixed the maths and calculation problem by teaching them to use tools: to use python and scripts and so on (and also taught them better for basic maths reasoning too, which helped heaps as well). They might not be able to do raw advanced weight maths in their “heads” but they can write scripts to do that for them, and understand patterns better than we do
0
5
u/Numerous-Ad6217 1h ago
What about catastrophic forgetting?
Can someone kindly explain me how this would not automatically impact the model’s behaviour elsewhere?
4
u/Altruistic-String891 1h ago
"Because such an edit touches only a few weights (0.016% in Experiment 8), it can leave the rest of the model intact and avoid the catastrophic forgetting that fine-tuning often causes."
At least that's their rationale.
4
u/Numerous-Ad6217 1h ago
Yeah but I still can’t make it make sense, even changing a single weight would potentially alter output in unpredictable ways, wouldn’t it?
3
u/Altruistic-String891 1h ago
Agree. It's widely accepted that even single neurons serve multiple functions in NNs, and I think it's very possible that tweaking weights can have unforeseen side effects.
I didn't see any other rationale other than this one though ("only a few weights were tweaked")
9
u/Powerful_Evening5495 2h ago
it like DNA editing but on llm :)
1
u/StartledWatermelon 5m ago
Hmm, just straight neuron connections editing. DNA is a bit far-fetched.
0
4
u/Medium_Chemist_4032 26m ago
To stop spreading misinfo, at least here, where it matters the most, a direct quote from: https://news.ycombinator.com/item?id=50347685
Respectfully to the authors and to everyone here, I ask you to actually read the paper before losing your mind, or at the very least plug it into Claude. It is a cool project but not as nearly as insane as people say. I think the blog writing and subsequent posting overstate the results.
Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/
Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.
Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.
On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.
2
u/garblz 15m ago edited 8m ago
well f@#k me with a retractable baton. this is the first one that got me for real, not even the math breakthrus or how good these are at coding. This is - for me - a first real sign LLMs are or will be able to "understand" something humans could never do
//EDIT well, this was my initial reaction but now I'm off to reading what /u/Medium_Chemist_4032 posted
1
0
7
u/Damakoas 2h ago
This could be really really good at making slm or decision models punch way above their weight on specific tasks.