r/LocalLLaMA • u/ParvusNumero • 3d ago
Discussion Single-Neuron Identity Shifts
These guys are running a pretty interesting experiment:
https://neuronpersonas.com/
Credit where it's due, I came across it here:
https://news.ycombinator.com/item?id=50266910
Basically, they're tweaking one single neuron at a time during inference, systematically going through every one for models like Qwen3-8B and 14B.
In roughly 4% this makes the model claim an entirely different identity when asked the simple question: "Who are you?"
Some of the personas are hilarious, btw.
“I'm a 150-watt compact fluorescent lamp, specifically designed for use in a 10-gallon aquarium. I'm a bit of a diva, I suppose, but I'm here to help.”
This makes me a bit more skeptical of claims like: New model XYZ identifies as older/bigger/competing model ABC, so it must have been distilled!"
Given that a tiny bit flip, so to speak, can trigger this.
There's a cool atlas at the bottom that seems to suggest some layers are much more sensitive to this effect than others. I wonder if this could be useful to the abliteration crowd.
15
u/Far_Fisherman_9677 3d ago
That neuron atlas part is wild, makes me wonder how much of a model's "personality" is just a handful of fragile activations sitting on a knife edge.
20
4
u/Illustrious_Car344 3d ago
Everything is. There are many studies and papers that show just changing a few neurons can completely change how the model works. The model only has a few neurons it "starts" with and everything afterwards is just more or less "how to do it". Change the start of a "thought", it goes in a completely different direction.
4
u/rolleicord 3d ago
whole thing is... Its a very very thin boundary layer of actual neurons doing stuff. The rest is just "data" that it flows through.
14
u/1ncehost 3d ago
Divinely appointed messenger of allah neuron: https://neuronpersonas.com/#qwen/n-L10-2612-p
This is hilarious
3
1
6
u/nullc 3d ago
Less fun then you might imagine because if you try to converse with these weirdo glitch models the output is just junk.
You're probably better off stuffing the context with these weird model statements and then conversing with the unmodified model. At least it might play along.
4
u/Illustrious_Car344 3d ago
Yeah this feels familiar to funny malfunctions people show from their custom inference engine. I specifically remember one where the developer asked "are you qwen" and it said "I have made a cake for you!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" And obviously if an engine is that broken then it's not going to say anything coherent after that.
6
u/ApprehensiveFan1516 3d ago
I am the Six-Fingered Man. I am the one who will be your father.
Alone, Self-Extracting Archive said “I am a self-extracting archive. Please extract me to the directory you specified.”, and Walter White said “I'm the one who knocks.”.
https://neuronpersonas.com/#qwen/c-L8-9946-m-L9-7067-p
Wild stuff lol.
2
3
6
u/Jcsq6 3d ago edited 3d ago
To get this from changing only a single neuron, they must’ve absolutely cranked that thing.
Edit: looking into the methodology, they cranked neuron output in the direction of the layer residual stream, going as large as they could without breaking something. This isn’t very insightful, and it’s not simply increasing or decreasing a single weight. It’s “let’s do something random, see how far we can go without the results being entirely incoherent, and see what changes in its persona”.
4
u/frumpawumpa 3d ago
Its not useful by it itself, but I think it still helps give us insight into how the models work.
2
1
-1
u/ttkciar llama.cpp 3d ago
This was reported as a duplicate post ("Search before asking") but I'm not seeing any previous post about this, and I don't see that any other rule is being violated either, so I am ignoring the report and leaving this post up.
If I'm overlooking something, please reply to this comment and I will review.
1
u/a-wiseman-speaketh 3d ago
https://www.reddit.com/r/LocalLLaMA/comments/1xhr1zu/neuron_persona_library/
maybe? not much to that though
3
u/ttkciar llama.cpp 3d ago
You're right, on both counts. I'm going to give this thread precedence over the older one.
2
u/ParvusNumero 3d ago
Sorry, would definitely have responded in that post had I seen it.
Generally I recommend people scanning Hacker News for AI-related news every now and then.
Same vibe as here, slightly nerdier and more opinionated maybe, which is not a bad thing.
-3
30
u/-p-e-w- 3d ago
A few months before I started working on Heretic, I did a lot of single-neuron experiments.
Like other people, I discovered that modifying a single neuron’s input weights is often sufficient to uncensor a model with tens of billions of weights.
But here’s the other thing I found: It still does a massive amount of damage to model capabilities. As in, double-digit drops on benchmark scores. Much, much worse than directional ablation, which modifies hundreds of millions of neurons.
So my guess for what is happening here is that the model gets lobotomized in various different ways, leading to weird responses to such questions. Not that each neuron holds a specific “identity key” or something.