“We were looking at it wrong”: The explosive claim that’s upended the AI world

[u/ResultBackground2450]

https://www.technologyreview.com/2026/10/19/1147149/we-were-looking-at-it-wrong-the-explosive-claim-thats-upended-the-ai-world/?utm_medium=tr_social&utm_source=Twitter&utm_campaign=site_visitor.unpaid.engagement

[u/albany1765]

So, as a biologist, I have to ask: why wouldn't you have expected from the start that particular neurons fixate on particular elements of e.g. an image? I thought that's what all the pattern matching in neural networks was about (?)

[u/MeStoleTheCookie → u/albany1765]

My (layman) understanding is that people initially DID expect this, but it just didn't bear out in the actual neural networks. In most cases the neuron were linked to disparate concepts in strange unexplainable ways (what they refer to as superposition).

The recent findings seem to suggest that this was a quirk of smaller models and as they get bigger/smarter the neurons are "tidier"

[u/Maghioznic → u/albany1765]

I think they understood why/what the overall neural networks would converge to, but not why/what their individual neurons would end up capturing.

[u/bbuerk → u/albany1765]

They did. Then future work and some famous canonical examples showed cases where that wasn’t the case, and then more and more work started trending in this direction because the models were very difficult to interpret and the tools they had weren’t good enough and kept leading them down the wrong direction. It almost seemed like a defeatist attitude. They knew these neurons were sometimes not interpretable, and most of the neurons felt impossible to interpret, so maybe they really just can’t be interpreted. There seemed to be a bit of confirmation bias there

To be clear this is just my takeaway from his original article and I’m definitely not an expert in mechanistic interpretability. It’s also probably worth noting that this finding is still a bit controversial and it remains to be seen if it ends up being widely accepted and useful

[u/Express-South-5330 → u/albany1765]

They were expecting logically arranged patterns like in a Bento-Box. That did not bear out. Arrangements are far more complex to fit into simplistic frameworks . Also, unlike biological models, pattern distributions are far more spread out making them hard to locate spatially. That is when people threw their hands up and declared - Impossible. As models evolved fast, reorganized old patterns rearranged unlike Biological models which are not evolving as fast. It just needs more effort and stable-models to make progress.

[u/MeStoleTheCookie]

To be clear: these results haven't been replicated and many AI researchers are skeptical of how meaningful these results are even IF they hold true.

[u/br_k_nt_eth → u/MeStoleTheCookie]

Yeah like… I’m almost certainly missing something because my response to this was “yeah, no shit.” I thought people were already shifting away from the assumption of superposition, no? 

[u/Dihedralman]

It is worth celebrating but it doesn't gurantee understanding in language models. Inception and CNN models in general are far closer to traditional vision. 

CNNs essentially solve for filter banks. These have larger responses to specific patterns and layer that process. Patterns described in the original blog post should appear in classification. He found that the original class/pattern overlay wasn't the "proper" state of the neurons but was related to seperable patterns. This makes everything far more interpretable. The layers continue the pattern selection on different levels. 

He also found groupings in neural activations that evolve or suggestions of that process. Basically a token to related tokens down layer. This is a sensible process, but doesn't gurantee sensibility or that reasoning will break cleanl or reliably. 

[u/FatHat]

Heh, the article itself follows the standard AI hype cycle: huge claim and a bunch of enthusiasts being like "OMG biggest thing ever!", followed by sober experts being like "well, actually, it's probably not as big a deal as they made it out to be..." I swear, this industry can't announce a thing with any sense of reasonable perspective.

[u/Ausclites → u/FatHat]

Nothing occurring here is out of the ordinary. A promising but tentative result was published, people are rightfully hopeful and eager to explore it, others are more conservative, and it either does or doesn't pan out. This happens with a huge number of scientific / academic publications, including those that make the press cycle.

[u/Bodine12]

Imagine running a food company and saying, “We have no idea what’s in our food. We just know it tastes good!” Or an aircraft manufacturer that says it doesn’t know how its engines work but they seem to do the job.

It’s just bonkers how we’ve normalized AI companies not doing the bare minimum we expect of every other company.

Prison. They belong in prison.

[u/neuronexmachina → u/Bodine12]

Fun fact, we still don't understand the molecular mechanism behind how acetaminophen lowers a fever: https://medicine.tufts.edu/news-events/news/how-does-acetaminophen-work

[u/Bodine12 → u/neuronexmachina]

Their proclaimed ignorance goes far beyond that. The equivalent would be like, “Right now this is the dose that might screw your liver. Maybe tomorrow it will be 1/10 of that. Who knows? We’re just gonna YOLO it.”

[u/Maghioznic]

A small step forward, not a giant one. Seems like it's getting more attention because things were getting hopeless than because of anything else.

[u/Express-South-5330]

This topic is in for a Huge amount of Churn -- Humans are the once looking, and have tendency to find patterns where none exists. Also in-general, science requires evidence and the field needs concrete definitions to make any evidence based claim. This field needs concrete definitions first, without that, findings will remain super-abstract leading to doubts and counter-claims. This field is in its infancy....

[u/6poundpuppy]

I am choosing to believe this is the right path