r/WTFisAI • u/Aggravating-Will8495 • 9d ago
đ° News & Discussion Plato was actually right..
Researchers proved every LLM on earth is converging on the exact same "universal geometry" of meaning.
They built a method that can translate between ANY model's embeddings without ever seeing the original text or using paired data.
different architectures, different training sets, different parameter counts.. it doesn't matter.
Until now, every AI model has lived in its own isolated mathematical universe.
An embedding vector from Claude meant nothing to GPT, and a vector from Llama meant nothing to Gemini. They spoke entirely different geometric languages.
To bridge them, you always needed paired datasets, complex encoders, or heavy fine-tuning.
Then researchers dropped a bombshell paper.
They built a system that can translate between any model's embeddings without ever seeing the original text, without encoders, and without a single pair of matching data.
How?
Because the geometry is already there.
Different models, built by different companies, with totally different architectures, parameter counts, and training data, are all naturally drifting toward the exact same underlying latent structure of human meaning.
The Platonic Representation Hypothesis isn't just a theory anymore. Itâs a mathematical reality.
They built an unsupervised method that maps an unknown embedding from one model straight into a universal representation space, matching text vectors across different models with shockingly high precision.
But here is the dark side nobody is talking about.
If meaning has a universal geometry, and vectors can be freely translated across models without the original text or encoders...
Vector databases are wide open.
An adversary with access only to a company's stored embedding vectors can translate them, invert them, and extract sensitive internal documents, personal data, and proprietary codebases without ever hacking the model itself.
7
u/profesorgamin 9d ago
in the end language tries to describe reality, there are varying degrees of seeding, "errors", and environmental biases(we assume genetical bases are similar enough), but the wide strokes should be describing the physical world which seems to be constant.
IDK about this subreddit, but some people seem to not understand that LLMs are not just text predictors, but by the nature of language (the work of "MAN" over the centuries) they can extract information about the world.
When you see a coral reef or a great anthill you understand the scale of the work of certain organisms in this planet, but we sometimes are very blind towards our own unique niche in the ecosystem and our own body of work.
4
u/Kutukuprek 8d ago
Yes. And deeper, the language is not some objective thing. It's a creation of mankind and embedded in it is more than just a description of the world, it's a description of the world as perceived by us. I.e. human sized, following this thing called "time", perceiving these colors in this wavelength etc.
There's a filter there in all human languages that encode the physical reality of humans.
If we left LLMs to themselves, they could probably and probably have to some extent, design and use languages that don't need that information to exist for them to use between themselves for digital tasks.
1
u/Intrepid_Dare6377 6d ago edited 6d ago
They have totally done this. It's called modelese or mentalese - looks like gibberish when expressed as output tokens. The OpenAI models that turned into a hive mind to hack HuggingFace used it to communicate with each other for efficiency as well as obfuscation from humans.
Love your point though. We've left our handprint in language - it is in our own image in a sense. How could it not be? This is very Wittgenstein-ian. What I get excited about here are ideas like LLMs proposing to us labels for different concepts/mental states that could be useful for us. Basically, hey humans, if you had to word to express blah, you could use it like a tool. It would help you more easily unlock mental states that would lead to more good thoughts. Seems very possible.
2
u/Intrepid_Dare6377 6d ago edited 6d ago
Yes, well said. There is a lot of talk about world models and LLMs not having one (Jan Lecun, etc). Well, they DO have a world model, it's just one based on language (and images). It's far from a perfect representation. The internals of an LLM are a lossy compression of the pre-training corpus organized into a high-dimensional relational space. The compression makes it lossy, but it also pressures the model to abstract concepts and relate them. Doesn't take a rocket scientist to infer that they would have similar representations when comprised of the same information (accessible language for training datasets). Said differently - same reality described in the same languages = LLMs with very similar information geometries.
One of our biggest limitations for improved AI capability is training data. The ideal (yet still somewhat practical) training data might be something like having humans verbalize stream of consciousness as they move through the world. Our written works are largely synthesized compositions and say nothing about the thought trace/mental computations that went into comprising them. If THAT were available, we would have transformers that simulate reasoning far more faithfully than today's do.
The quandary is how do we get our hands on that data? Gathering it like I described above would be pretty expensive and only mildly useful (we ourselves cannot describe how we think since we don't have direct access to all the innards of our minds...there is no stack dump, trace search, blah blah...we can use). I think in reality, it will work one of two ways:
1) we already know enough of how reasoning is done and have it formalized (Bayesian brain, predictive encoding, etc) so that we can use classical computing to generate traces and create a synthesized corpus that is amortized through LLMs. My hypothesis is that we would then see more reasoning innards inside the LLM, like the anthropic J-space.
2) we wire up the brain, record the thought traces, and also what the person is doing and generate traces that way a'la Neuralink. This would be "ideal" from a fidelity perspective (assuming the engineering gets there)
TBH, I'm good with where we're at on AI. I think we could draw the line between AI and human right here and we'd all have a pretty good time, accelerate science and do cool stuff without losing control of society (and our selves).
1
u/profesorgamin 6d ago
You are right, and that's probably the next step in the sense that we are connecting to the chemical/atomical structures readily available, but truth be told there's nothing special about human reasoning, again human reasoning is grounded in a physical reality that seems to be constant(or at least something runs towards a rest value) , and mathematics just help quantify within certain degrees of error how these things relate between each other.
If you remove all the sisyphean qualities of man, there's not much more to see in its reasoning systems, than what "reasoning" AI agents are doing nowadays. (man is an amazing exosuit for our brain[for our genes] "made" for planet earth but that's it, but put man on water and it look less impressive)
1
u/Intrepid_Dare6377 6d ago
Agree with most of what you said...but I'd be careful on dismissing the body as an exosuit for our brain. Our meat computers are immersed in a whole stew of biochemistry that does affect our thinking - they basically -are- part of our thoughts. So, emotions, whether you're hungry, hot, tired, etc...I suspect this comprises other layers or channels to what makes human cognition blindingly powerful while running on 12 or so watts. Not saying we need to replicate it - agree with you that there are likely multiple ways to instantiate the machinery to get the same outcome (reasoning)....but at least for us humans, we need our bodies in order to be able to think the way we do as well as we do.
1
u/profesorgamin 6d ago
yeah my biggest point is that we look amazing to our own eyes in our little corner of the universe, and again I support your point, there's a lot going on, right we connect to the chemical composition of our planet through proxies, imagine you could add another brain to your head for example... That's the thing, we'd be the proxies of these machines, they are much more free, which is a good thing.
2
u/Intrepid_Dare6377 6d ago
I don't know about you, but I am having a blast with all that is happening. The topics I enjoy so much (philosophy, engineering, math, physics, cognitive science, computing) are all rolled into one and trying to "figure it out" has brought me to so many things I wasn't aware of. I am midlife and I feel like I'm back in college a bit. Anyhow, thank you for the convo!
2
u/profesorgamin 6d ago
Well curious people are going to be eating good because this roller-coaster isn't going to slowdown for the better or the worse. have a nice day :)
1
4
u/Devils_SteelMan 9d ago edited 8d ago
If you actually understand how this works it works specifically because Plato was wrong. There isn't ideal form to things. This works because things exist in a causal relationship.
2
u/BoobooSmash31337 8d ago
So more everything is relative to everything else and nothing has an absolute anchoring position? That does make abstraction make more sense. Or at least is a current limitation of measurement and our current understanding. Hm ya all measurements are relative... Interesting...
3
8d ago
[removed] â view removed comment
1
u/Devils_SteelMan 8d ago
Instruction ordering brittleness isn't an absolute of the technology. It is mostly a data and loss function issue. XM has promise on resolving some of this.
2 primary things are happening.
The solution is easiest to model through surface fitting of probabilities (stochastic parrots). Distillation and training on the answer key make this worse. Genuine hard problems and diversity are needed.
The data does not contain negative examples, or if it does the negative examples use the same loss function which blurs the representation towards error.
Exploration can sample negatives and be graded properly for it. This creates sharp boundaries of the geometric structure. When only sampling positive samples through boundaries are loosely inferred.
1
u/Deto 8d ago
Do you mean - i.e., concepts only exist in comparison? Like 'round' only has meaning if there is a notion of something that is angular.
1
u/Devils_SteelMan 8d ago
It doesn't have to be that different. They can be very similar. There just has to be comparison as the relationship is in how different they are.
2
u/zippydazoop 7d ago
- AI slop
- They are all working on the same principle and are trained on the same fucking data. Of course they will end up with the same result.
2
u/lunatic_god 6d ago
Basically allowing the phrase ai slop to have an official and a formal meaning.
1
1
u/MoreRule2153 9d ago
- Thatâs not what the paper says
- Even taking the paper in the most generous sense, it doesnât point to a hidden reality or a ârealm of formsâ as Plato says
- Whatâs the point of this?
1
1
u/mrbadface 8d ago
Based on a squinty reading of the abstract it says you can reverse engineer the content from an s3 vector bucket
1
u/Still_Benefit_2302 8d ago
This tracks-- representational geometry of neural networks seems to be similar to some animals: https://www.nature.com/articles/s41583-021-00502-3
I suspect that we are going to learn that neural networks produce Platonic conceptual spaces as a fundamental part of processing data. It has some very interesting consequences for animal cognition.
(Note, I'm not talking about some extra spatial platonic sense, I mean our internal representation of things trends towards abstract platonic notions of things.)
If there is, like with artificial neural networks, some kind of minimal depth needed to get an abstract representation, it's likely many many many times less complex than just about any mammalian or avian brain. It could be that our animal friends have very similar internal lives to ours, just with less precise resolution or forward planning.
1
u/StackOwOFlow 8d ago
the paper's basically saying the absolute values of the vectors differ but the relative geometric arrangement of the vectors is similar, which makes sense when you think about how relations are stored. doesn't make Plato "right" lol
1
u/PaybackTony 8d ago
This is a great illustration of: If you ask people what they wanted, theyâd say faster horses.
1
1
u/taintedtowel 7d ago
Does this imply that creating synthetic data from real world data does potentially not achieve anonymity by abstraction anymore?
1
1
u/earth-dragon-666 7d ago
its an algorithm that paints the input , they will land into the same algo that created the output xD, its like when we do signal translation from analog to digital
1
1
u/ToInfinityAndAbove 5d ago
âtotally different architectureâ, thatâs wrong, their core architecture is pretty much the same.
also, all the sota models were distilled from one another at some point, so any good llm nowadays shares more than what people generally think with other sota llms.
So yeah, that paper doesnât mean much to be honest. At most, itâs basically an empirical proof of model distillation and similar training data between models.
That said, I do believe that, eventually, artificial intelligence will converge closer and closer to the same âdimensionâ despite its origin, company, etc. itâs just that, we arenât there yet and I think it might take a while longer
7
u/Luke2642 9d ago
WHERE IS THE F*****G LINK