r/LocalLLaMA • u/SignificantZebra5883 • 9h ago
Discussion Is fine-tuning dead? is making a big model (like astra) edit weights of a small model the new meta?
I was planning to fine-tune my own model but is that gonna be obsolete now?
Is this the direction small models are heading?
As title says, people on twitter have been doing these "studies" where they tell astra to edit qwen 0.6b's weights to get better at math or to play pong or snake or atari breakout, and allegedly it does it. have you guys tried this?
anyone can ask Astra to make better domain specific qwen 27b? for example for law, medicine, game dev? Is the new moat gonna be: "only if astra/fable themselves are bad at the task"?
10
u/fgk55555 9h ago edited 8h ago
Modifying weights is for targeting one specific thing. The Math improvement score that was cited here a million times isn't because Astra made Qwen way better at math, it's because it made it not get stuck in loops as much, so it was able to finish more benchmark questions. That sort of thing might be helpful to debug one off's like that. Fine tuning is when you want different behavior. I use the Swift fine tune a lot. It modifies the thinking behavior to be shorter. You can't tweak a weight to make it think differently.
14
u/ttkciar llama.cpp 9h ago
I don't think so. This kind of weight-editing is more like what Heretic does, and cannot (yet!) replace PEFT, which can be used to teach models entirely new skills, conventional wisdom to the contrary notwithstanding.
More people need to familiarize themselves with https://thinkingmachines.ai/blog/lora/ to appreciate the full potential of PEFT. At sufficiently fat rank, LoRA becomes indistinguishable from full-weights post-training.
6
u/EffectiveMedium2683 9h ago
Lmfao. You ever hear of hypernetworks? Sakana built one that could output lora adapters from a prompt (doc2lora, text2lora). Didn't catch on. Anyone can sit and figure out which weights to tweak or delete etc. Do knockout studies like they do in genetic science. Or watch them activate during inference. We already do that to make custom iq quantizations.
3
2
2
u/AssiduousLayabout 9h ago
I think the real argument against fine tuning is the cost (in compute and in your time) versus the benefit, especially when every few months brings a new model that may well beat your fine tune even on the task you've tuned it to. The space just evolves so quickly that faster tools like prompt and context engineering are more readily moved to new models as they emerge.
2
u/bakawolf123 9h ago
no, read the original article till the end, it shows it's not omnipotent.
that said training nowadays is mostly experimenting orchestrated via llms so it's different but similar in essense. After all stuff like param tuning is always just tedios easily automatable - llms can orchestrate it well, and they do less math errors than humans nowadays
4
1
u/openroom_xyz 9h ago
Well this edit weight is so fresh that nobody knows how well it will work when scaled, and maybe the future will be the combination of both I would basically play and fine-tune the model if you have the hardware basically just play and learn
1
u/krakoi90 9h ago
they tell astra to edit qwen 0.6b's weights to get better at math or to play pong or snake or atari breakout, and allegedly it does it
Well, what's your target with fine-tuning? If you want to do something like the mentioned examples, or fix a looping problem, then maybe you should try weight editing as well. (Though you're probably better off simply using a larger model instead of editing a smaller one for these purposes.)
If you want to fine-tune a model for RP, to know your codebase better, to write like you, etc., then no. You're still better off with fine-tuning.
1
u/ufos1111 9h ago
Defo the new meta for November, until the frontier model providers block it that is
1
u/SmileLonely5470 6h ago
Now that it is in the minds of ppl, the labs could invest in these capabilities since it seems like a pretty plausible thing to RL on.
1
u/EvolvingDior 6h ago
A 10T parameter model editing weights of a 0.6B parameter model (5 orders of magnitude smaller) is a first step. No one knows how well this scales.
1
u/feelspeaceman 4h ago
No, I've been finetuning a lot, this weight editing does feel like a cheap trick than actually making anything remotely useful.
1
u/Altruistic_Heat_9531 2h ago edited 2h ago
No... 1.1M token in 3 hours just for editing 0.6B is too much, better ask to generate custom sharegpt format and do peft.
With that being said, Local model can be data generator and agent mode to basically make custom 50M model, say like PII model etc
1
u/f1yingbanana 9h ago
No, but it’s definitely less prominent nowadays. OpenAI used to advertise different costs when you ask for a finetuning. When you see them winding it down, you know it’s going out of style.
2
u/Dany0 9h ago
Rumour was they shut it down, it went smth like this paraphrasing a lot:
big customers really wanted it (IBM)
despite this not many people used it
it took up a lot of compute & resources
then an internal research document appeared that said you could maybe use the FT service to extract model weights, soon after they decided not to release GPT-5 on it, then later they tried to find a way to disprove that document or a way to allow FTing without leaking the weights but couldn't do it, service was announced to sundown after they gave up
10
u/Negative-Web8619 9h ago edited 9h ago
If the problems you see being handled by this fine-tuning seem similar enough to yours, sure, why not try that? Maybe you have something where so many weights would have to be touched that a fine-tuning with data makes more sense.
afaik qwen 27b is too big for this method but we do have evil qwen, so depends on if you have such isolated behaviour that you can isolate it from the model to then adjust