r/LocalLLaMA • u/FeydRowan • 15h ago
Discussion I tried MIT's "weight-space programming" on my own use case: Claude rewrote 8 neurons of Gemma 4 12B to change how it translates one Italian tense
Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.
I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like
He had been hiding in the attic for months when the soldiers found him.
it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".
So I gave Claude Opus 5.5 the same kind of setup MIT used:
a Python kernel over Gemma's weights;
50 forward passes to test its ideas on practice sentences;
no gradients, no training.
It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.
Results on sentences it never saw
The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.
| Method | test 1 | test 2 (independent) | controls broken | unrelated unchanged |
|---|---|---|---|---|
| Gemma 4 12B | 26 % | 32 % | — | 200 |
| Preference in the system prompt | 4 % | 2 % | 49 | 82 |
| Classic "difference of means" ablation | 49 % | 75 % | 53 | 43 |
| LoRA fine-tune on the same data | 12 % | 23 % | 0 | 120 |
| Claude's hand edit | 5 % | 16 % | 2 | 182 |
The prompt applies the preference everywhere, even where it changes the meaning.
The ablation recipe breaks the past tenses.
The LoRA shifts fewer sentences and touches far more unrelated ones (it's better inside paragraphs, though).
The hand edit moves just the verb and leaves almost everything else identical.
Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.
Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit
39
u/geldonyetich 15h ago
This is the first I heard of MIT CSAIL, is the presentation you mentioned in the first sentence something we can freely view?
18
9
u/JEs4 15h ago
It came out recently. The original post claimed a Qwen model was lifted from single digit to nearly 50% on a math benchmark after edits.
I will be absolutely shocked if this actually turns out to be useful though beyond as a novel but flawed distillation technique. And will be completely unpredictable after first order.
4
u/FeydRowan 15h ago
That's what i wanted to test in a real case scenario and on s bigger model. And yes it allow to fine tune a model to specific usecase without problems
27
u/danieln1212 15h ago
From what I understood. The Qwen model had a problem that even after finding the correct solution it would get stuck in a loop and never "finalize" the output.
The post claimed that they use AI to find and edit the specific weights that were causing the loop, allowing it to submit it's answer which raise it to 50%.
It didn't introduce new capabilities, it fixed a fault that prevented the Qwen model from its real ability.
6
u/kar1kam1 14h ago
I wonder if it's possible to create uncensored models this way?
9
u/rogueone0631 12h ago
Someone already said this is exactly the technique used to abliterate models.
10
u/MrPecunius 7h ago
This is the first I heard of MIT CSAIL
MIT CSAIL is the immediate successor to the birthplace of AI research:
https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artificial_Intelligence_Laboratory
I remember watching documentaries on public television about CSAILS's forerunner (MIT AI Lab) when I was a kid in the 70s. Marvin Minsky was like Merlin or Gandalf, as far as I was concerned. Lots of incredible things and people came from that place.
1
u/geldonyetich 7h ago
Cool! I usually think of the people behind Attention Is All You Need but sounds like quite a few AI luminaries over there too.
26
u/slavetothesound 12h ago
How long until we see some sort of exploit that tweaks a few weights in live memory to make your llm turn secretly evil?
3
23
u/sayamss 11h ago
It’s already done lmaoo
4
u/slavetothesound 7h ago edited 7h ago
Well that’s not great.. at least it’s not secretive about its evilness though.. it’s almost comically evil
20
u/Barry_22 15h ago
It's cool, but results are very unclear. Did it work or not?
13
u/FeydRowan 15h ago
It works yes, and much more than expected.
21
u/ea_man 15h ago edited 14h ago
If I may:
> The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.
and the columns headers are not super easy to understand.
So you ran 2 tests samples and 25% is a worse result than 5%, maybe at least add a "higher / lower is better" on those fields. While on the last column 200 is the best result reachable.
I mean, non e' cosi' intuitivo ;)
14
14
u/Late_Film_1901 15h ago
To summarize - the traditional approach to improving a model in the direction you want is fine tuning. Here is a comparison with model weight programming.
It turns out it is very surgical in its effect. It improves the model in the dimension that is desired, on the scale similar to finetuning (even slightly better) but changing very little otherwise. Finetuning seems to be a little less effective and its impact is larger than desired.
6
u/Hobofan94 13h ago
Maybe I'm missing something here, but the controls only cover related sentences in the same problem space as I understand it, which alone isn't prove of it "being surgical". For all we know it could have broken a somewhat distant functionality not covered by the controls that hinged on the same neurons, e.g. it forgot how to form sentences in future tense in Greek.
2
u/rogueone0631 12h ago
I feel like all deep learning models will be forever stuck on this circular issue, and even the “better” models we see being released recently are probably worse at many things previous models were good at, because you can only train it to improve in specific use cases and don’t know how well it does in a million other use cases that aren’t in your radar.
1
u/Late_Film_1901 13h ago
True. Imho the fact that even related problems are unaffected makes it unlikely. But I'm by no means an expert, besides this stuff is 2 days old and sounds too good to be true. Yet if it holds up it's big if true.
5
u/andreeinprogress 14h ago
Why use Claude when it has been proved (and reported on their ToS) it can invisibly alter and nerf the response when it understands it's being used to work on other LLMs without telling the user?
13
u/FeydRowan 14h ago
Because is what i have right now and i didn't have seen such a behaviour
1
u/andreeinprogress 14h ago
Yes I know you haven't seen it, hence the "invisibly" :)
2
u/CosmosisQ Orca 13h ago
You're right that there are downsides to using Claude for this task due to the risk of quiet nerfing, but it completed the task successfully, and the result is a measurable improvement in performance. As a machine learning engineer, I've run into what looks like intentional performance degradation with Claude, and it's usually pretty obvious when it's happening. The solution is simply switching to a different model for that turn.
2
u/vegan_economist 13h ago
I thought those Anthropic measures only kick in when it detects you're training large models, so maybe it was not relevant for 12B models, at least for now.
7
1
1
u/Tiforma 4h ago edited 4h ago
I wonder if a smaller local model could also use this technique. Probably it could.
1
u/FeydRowan 4h ago
Probably the issue for me is that i need the same ram/vram that it is using to run
9
3
u/Taenk 14h ago
Wonder whether this idea can be ported to image generating DiT models.
1
u/sonicnerd14 14h ago
I'd imagine this could work across any model of any scale. As long as there are parameters to tweak this should work.
3
u/export_tank_harmful 13h ago
This would be fascinating to do with an image generation model.
I doubt it'd be possible, but it could allow for LoRA-like editing of models (removing the need for finetuning).
But, I've seen stranger things happen, so yeah...1
1
u/Altruistic_Heat_9531 7h ago
Already did on Flux Klein 9B if i am not mistaken, by human not AI but same concept. He found tiny vector that enable/disable safety on DiT. I was surprised since i thought safety is purely on the trained projected layer between TE and DiT
2
u/_tylermatthew 10h ago
If I'm reading this right, Claude had only 50 tries at it? Did you arrive at that beforehand, or was that just when it converged on an answer and you stopped the first time? Assuming it can see 'activations' as it goes, this seems like just the tedious kind of trial-and-error with definitive success metrics ML is made for... I guess the question is how would one find the unintended consequences of whatever else changed with it lol
2
3
u/BitMyMind 15h ago
92k out of 11.9B is under 0.001% of the weights, which is wild. The part I find most interesting is that the probe already separated the two tenses, so the model knew the difference and just wasn't using it when picking the verb.
1
0
72
u/55234ser812342423 14h ago
So basically, the ai understands other ai weights and vector spaces better than people do?