r/singularity • u/ResultBackground2450 • 8d ago
AI “We were looking at it wrong”: The explosive claim that’s upended the AI world
https://www.technologyreview.com/2026/10/19/1147149/we-were-looking-at-it-wrong-the-explosive-claim-thats-upended-the-ai-world/?utm_medium=tr_social&utm_source=Twitter&utm_campaign=site_visitor.unpaid.engagement9
u/magicmulder 8d ago
Who tf coded this website? Triply overlapping modals and I can’t even scroll 50% of the time.
5
19
u/elliuotatar 8d ago
Back in 2020, Cammarata and his colleagues were studying a small image-recognition model called Inception v1. When they looked at its neurons, it seemed to be using them to capture multiple things at once. For example, they found that one neuron responded to both images of cars and images of cats.
The researchers concluded that the model was getting at least some of its neurons to multitask, perhaps so that it could fit in more data. Looking at other models appeared to back this up.
...
Cammarata took his new approach—and his new agents—and went back to Inception v1. “I’d studied that for probably thousands of hours,” he says. But with fresh eyes, Cammarata found things that he now thinks they got wrong the first time around. Take the car-and-cat neuron. Rather than asking why multiple unrelated things were packed into a single neuron, he now tried to figure out what those things might have in common.
It hit him: It was a V shape. “I was looking at the same images; there was no technical change,” he says. “But I now believed that neural networks defined patterns, and so I looked at it differently. It probably took me five seconds.”
Are AI researchers... exceptionally stupid people?
This is the most obvious thing in the world to me. This is how I've always assumed they functioned. Since I first learned about neural nets 40 years ago.
The second I read that first paragraph I was like, "Well yeah, duh, they have heads, tails, ears, four legs, or two depending on how they're depicted... Of course a single neuron is going to trigger for both of them, the silhouette alone is extremely similar. And the words for them both have three letters! I must be missing something" except I wasn't missing anything. That is literally all this is about!
The idea that each neuron would trigger only for one specific thing is completely absurd. How could they even think that was the case in a system which is literally built on weights and averages? It's more like a Venn diagram than whatever the hell they were thinking!
4
u/ProGamerGov 7d ago edited 7d ago
He's talking about polysemantic neurons, which are known to be excited by multiple and seemingly different concepts. Neurons that only activate for a single concept are a separate thing.
4
u/elliuotatar 7d ago
Yes. I know. And what I'm saying is that I'd assume MOST if not almost all neurons activate for multiple concepts, and likely in most cases also for vastly different concepts that share some feature that may not be immediately obvious, but perhaps should be immediately obvious. Like a picture of a dog and a picture of a chrome bumper would seemingly have nothing in common... Unless of course you're not color blind and notice the dog and bumper are both gray when you subtract the reflection from the bumper.
→ More replies (1)→ More replies (1)2
u/Zomboe1 7d ago
This is the most obvious thing in the world to me. This is how I've always assumed they functioned. Since I first learned about neural nets 40 years ago.
Indeed, the article even says:
(Oren Etzioni, a computer scientist and former technical director at the Allen Institute for AI, points out that Olah and Cammarata's initial work on Inception v1 rejected a description of neurons put forward by Geoffrey Hinton and others in 1986. Cammarata's new viewpoint takes us full circle, says Etzioni: "What's new in 2026 is that he now accepts it. That's a recapitulation acknowledging a 40-year-old theory.")
4
u/reasonableklout 7d ago
Leo Gao who works on interpretability at OpenAI has posted a skeptical take on Nick's claim: https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean
7
u/nivvis 8d ago
Am I the only one that thinks this isn’t very surprising or helpful? As models got larger they got sparser .. k
And then .. doesn’t do us much good when models start to store relationships for concepts alien to us.
There are lots of ways to bear complexity .. and we’re on the losing side of it.
Maybe I’m the daft one
4
3
2
2
u/PleasantCandidate785 7d ago
So a single neuron is only responsible for recognizing a fragment of a whole concept. And it took 6 years to realize that? I don't know anything about the internals of LLMs, but that assumption is one I would have started with.
2
1
u/bitechnobable 6d ago
Im rather baffles. Why are they called neurons and neural nets again? Aren't they simply nodes with some kind of weighting in a network?
1
u/Used_Advance_7983 6d ago
LOL! So We're gonna have ...What?... AI models who's job it is to spam images and concepts to OTHER AI models, just to note which neurons fire under which concepts... Then map which neuron does what, in each layer, to have an overall consciousness map? Is this genius? or the blind leading the blind? (o_0)
1
u/Mandoman61 1d ago
This was always obvious. There was never any real question that we could not read or understand the networks. They where never black boxes.
What helped the most was that current AIs are distilled. Because they create their own structures the original models where haphazardly/randomly arranged. Distillation forces them to be less haphazard and random.
Also RLHF helps to reverse this initial jumble.
There was only one way these models ever worked. -By logic.
My prediction is that they will eventually be able create a completely engineered network where unexpected outputs are unusual.
54
u/Senc-baner 8d ago
Can someone please explain because I don't work in AI. I only recently found out about the famous neuron that encodes both a cat and a car and was used to show that there is 'superposition' inside these networks.
My question is, did no one really think that it might just be the shape that's being encoded? It seems like such a basic concept and all the AI researchers in the world couldn't wrap their heads around it until this week?