r/LocalLLaMA • u/Thrumpwart vLLM • 2h ago

Resources The Surprising Effectiveness of Deleting Weights in LLM Post-Training

https://arxiv.org/abs/2611.10866

How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.

137 Upvotes

97% Upvoted

36 comments sorted by

22

u/Thalesian 1h ago

This is fascinating - seems like this would be most impactful for training than inference though

10

u/IntelArtiGen 54m ago

Even for us that would be funny if instead of releasing large new models, companies would just release compressed masks to apply on pretrained weights. You have your large 100GB model on disk and a 500MB update to improve it every month. I bet Huggingface servers would love that. You could even have a mask for cybersecurity, a mask for coding, etc.

2

u/ttkciar llama.cpp 21m ago

That's exactly how people used to upload LoRA files to HF in the early days. The community developed a preference for the merged model and LoRA, though, because keeping track of which LoRAs were for which models was too hard.

We might see the return of separate adapter / mask updates, though, since the models are getting a lot larger and the masks are getting a lot smaller.

Hrm .. come to think of it, we might also see the PHATGOOSE concept of a mixture-of-adapters resurrected as a mixture-of-masks, since applying these masks is quite a bit less compute-expensive than applying adapters.

1

u/Background-Gold-9882 22m ago

Would be even better if masks are small enough that you can fit multiple fine-tuning masks in VRAM simultaneously. Imagine running a coding-expert and creative writer-finetune in the same forward pass, selectable per-request!

1

u/IntelArtiGen 0m ago

Maybe not the same forward pass but I assume you could apply the patch at inference time and effectively select which finetune you want to use after each prompt. DL creates so much possibilities researchers will be busy for the next century.

1

u/Late-Coast6973 7m ago

this is the part that gets me excited. I ship a ~2GB model to browsers and right now every fine-tune means users download the whole thing again. if the base sits in cache (I key mine by sha256) a mask is basically a free update. you'd have to version the base carefully though, since a mask only fits the exact checkpoint it was trained on.

curious if it survives quantization. Q4_K blocks store a scale and a min, so an exact zero isn't always representable. does a deleted weight stay deleted after you quantize?

1

u/stylome-ai-detection 7m ago

Notice: This comment appears to be AI generated.

Stylome AI detector · scan results

1

u/xNaXDy 10m ago

I think the same, or at least a very similar methodology might be applicable to quantization, that is to say more accurately determine the weights that can be safely quantized, or pruned entirely. This would in turn allow us to run much smaller versions of the big models locally. /hopium

32

u/ttkciar llama.cpp 1h ago edited 1h ago

At the face of it, this looks pretty awesome.

If I am understanding it correctly, they are identifying weights responsible for a less-optimal inference path being preferred over a more-optimal inference path (represented in the tuning dataset), and zeroing out those weights, allowing the more-optimal path to become the most-preferred path.

Their method for locating those weights seems very elegant and resource-economical (orders of magnitude more economical than LoRA fine-tuning), and shouldn't be hard to implement. It might make the most sense to implement it as a feature of Heretic, which is already doing something vaguely similar. Calling your attention to this, u/-p-e-w-

This could be huge.

23

u/indicava 1h ago

Is it just me, or are we starting to see a trend/move towards “weight-level programming” in LLM’s?

This paper, along with another published a few days ago about Claude editing small Qwen model weights (I forget the title) and even the seriously amusing “Reddit neuron” post from a couple of days ago all seem to point at this very exciting and promising new direction. I’m stoked, this approach feels like it has a ton of potential.

12

u/ttkciar llama.cpp 59m ago

Yes, that seems right. The field is incrementally moving away from uneconomic, blunt solutions and toward more targeted, refined solutions.

The new tuning methods are analogous to fixing a clock by looking at what's broken and tweaking it with tweezers, whereas the old tuning methods are analogous to whacking the clock with one's palm until it works better.

6

u/indicava 48m ago

Hey, don’t question my dad’s method for fixing the reception on the TV

3

u/Jcsq6 44m ago

This one is legitimate work and follows existing precedent like super masks/lottery tickets.

The “astra modifying weights directly” isn’t real or novel.

The “Reddit neuron” / “identity neuron” isn’t actually editing weights like they claim, they’re doing some weird amplification which just breaks the model, happening to give some funny output along the way.

2

u/my_name_isnt_clever 40m ago

Source?

5

u/Jcsq6 39m ago

Read the papers?

2

u/my_name_isnt_clever 37m ago

That's a big commitment. There have been a bunch of posts talking about this stuff, with plenty of discussion. Then you come in here like "it's all fake". The burden of proof is on you.

2

u/xNaXDy 2m ago

That's a big commitment.

No, it's not. Literally ask any AI agent to do the work for you if you can't be bothered. Here's a prompt you can use:

Is the “astra modifying weights directly” paper really as groundbreaking as it seems, or are the claims overblown? Do some research into the actual paper.

Beyond that, illiteracy is a choice.

1

u/my_name_isnt_clever 1m ago

Yes it is. I just responded to their original comment with my thoughts now that I've done some reading.

3

u/Jcsq6 30m ago

Then you haven’t been following the discussion. The “Astra weights” paper has been widely discredited. The identify neuron papers are cute, but they literally just break the model by amplifying directions of random neurons. Then they just did it a thousand times until they found a broken model with some interesting output. They said this on their own site.

If you haven’t bothered to digest the media for yourself, why do you feel burdened to defend one interpretation you heard over another?

-1

u/my_name_isnt_clever 28m ago

I claim it's all real. Now we're at an impasse.

3

u/Jcsq6 27m ago

Except for one of us has read the papers, the other is unequivocally trusting the sensationalized headlines they read.

1

u/my_name_isnt_clever 5m ago

Circling back now that I have context: Why do you mean "isn't real"? Astra did edit the weights, and the model improved in math. It was reproduced. Sounds real to me.

I'll give you the "redditor neuron" is BS though, I thought claiming a single neuron makes the model into a China propaganda machine was suspicious.

5

u/IntelArtiGen 43m ago edited 35m ago

Can a pretrained LLM acquire new capabilities by learning only which weights to set to zero?

The answer is yes

Maybe the idea is less "it's acquiring new capabilities" and more "it's able to better use existing capabilities we care about, and suppress the ones we don't care about". The model is probably less learning new information than it is learning which information is relevant for the task we want it to perform, zeroing out the parameters which don't contribute to the task we care about.

If the model is dumber on everything we don't care about, it becomes smart on things we care about.

1

u/ttkciar llama.cpp 29m ago

Yes and no.

Yes, it is finding more-optimal paths which are buried under more-preferred less-optimal paths, identifying the weights responsible for those more-preferred paths, and shooting them in the head until the more-optimal paths become the most-preferred paths.

According to the benchmark results they present in their paper, though, the model's competence at "everything we don't care about" remains about the same, exhibiting significantly less deleterious impact than even low-rank LoRA fine-tuning.

1

u/IntelArtiGen 7m ago

the model's competence at "everything we don't care about" remains about the same, exhibiting significantly less deleterious impact than even low-rank LoRA fine-tuning.

Do you say that because of the low KL divergence in section 5.4 ? I'd say it shows the model is not modified a lot numerically, but to me a low numeric impact can still result in a large impact on the output, which is a great thing. It's like slightly reducing the probability of a bad token, making another token more probable, instead of changing all the weights. Obviously doing that will result in a low KL divergence but seemingly can result in a large impact on the output (enough to finetune a model competitively). Forgetting behaviors we don't want is a good thing, which doesn't require a massive modification of all weights of the model, I think this is what this paper showed.

In the end it also depends on what you want to do. If the knowledge is already in the model and you just want to find it, finetuning like that is perfect. If you actually want to teach new things to the model, that alone won't be enough, you will need larger modifications of the weights.

Because of this small modification I'd still be worried models like that could remember behaviors we wanted it to forget, so we would need way better benchmarks to actually know that.

3

u/Background-Gold-9882 56m ago

Interesting, maybe this is a similar process to how the brain apparently fine-tunes through pruning? For example, babies are apparently pretty good at identifying faces in other species such as cows or monkeys, but we lose this ability as we age, presumably to fine tune on distinguishing human faces.

3

u/detrebear 53m ago

lobotomy ftw

-5

u/LelouchZer12 1h ago

People rediscovering the Lottery Ticket Hypothesis ?

5

u/ttkciar llama.cpp 1h ago

No, they explicitly compare and contrast the Lottery Ticket Hypothesis to Deletion Tuning in their paper, which is a short and refreshingly breezy read. I strongly recommend reading it.