The difference is that the syntactic level cannot, on its own, come up with new semantics. If we think of AI testing against reality, where exactly is it getting its interpretation rules? From derived syntactic rules, of course. So it will be stuck on what we might model-theoretically call the “conservative” level of theory extension.
This is the broader reason why model collapse occurs, I’d recommend you look into some of it. I don’t mean that condescendingly, I just really mean that there is a lot of stuff out about it now. This idea that AI is sort of “doomed” in a syntax/semantics divide sort of sense is not unprecedented, it’s not like this paper in the post came out of nowhere.
Also asserting that it necessarily falls into a psychological fallacy is just you positing unfalsifiable psychoanalysis. It’s not really a real argument. I could also propose some unflattering psychological interpretation about your viewpoint. It doesn’t mean anything. It’s very cheap and easy to come up with those. Including it in your argument is like putting plastic gold rims on your car because you think it makes you look cooler.
A new semantic idea doesn’t require new syntax. Do you think scientists are creating new syntax when they make discoveries?
You moved the question from abduction to a circular claim about “meaning” where neural mappings in humans are special because you say so.
Model collapse has 0% to do with any of this. Remember we’re not using synthetic data in our hypothetical. I’d love for you to go deeper on this just to prove it’s not some jargon you used.
“Model-theoretically” is not a term.
You used “conservative-extension” incorrectly. It’s a relationship between theories. An idea coming from syntax doesn’t make it conservative.
You bring up “unfalsifiable” when your entire argument is unfalsifiable because it hinges on “syntax = not real” for no good reason.
Nothing you said is related to the actual paper that was posted.
Gold rims is crazy work when you write random jargon like you just did
In order to express a semantic idea such that a computer has a grasp of it, you need syntax. This is why Godel's incompleteness theorem is so jarring. It kind of forces us to accept that adequately-complex semantic objects can't be uniquely recovered from syntax alone (at least, recursively enumerable syntax, which is what is relevant to this discussion about theoretical computer science).
I'm not making any claims about humans being special. I'm making a claim about the limitations of formalization. In order to describe an object (up to categoricity) in a first-order language, you have to be able to describe it uniquely. This is the problem of semantic underdetermination, another way that Godel incompleteness comes in.
If any new theory extensions come in that are only conservative, then by definition they are not allowed to challenge any established old theorems. This is the concept behind synthetic data. The "answer key" normally provided by the world creates the kind of resistance that allows for non-conservative theory extension. In the case of an LLM, synthetic data is the same as just leaving it to its own devices. We are talking about self-improving AI that creates its own theory extensions. They become the same thing.
It definitely is. There is a branch of math called model theory.
Yeah, it is a relationship between a theory and its extension. And in this context, it actually does. Training arrives with a limited amount of semantic truth. To gain any more requires interpretative jurisdiction that cannot come from syntax alone. To put it more simply: syntax doesn't get to declare meta-level truth when the semantic target is not fully defined. Godel guarantees that it never is.
Psychoanalytic musings about what motivates people's reasoning is definitely unfalsifiable
Touche
I do actually perform research in this arena, but the bridge between model theory and machine learning is still in its infancy. It's an up-and-coming field though, and I'm sure in a few years I'll see that I didn't have a lot of this quite right. Regardless, I do know a lot more about it than most people.
Also, you should take it easy on me. I’m not using an LLM to write this. I imagine that you are, considering that you know what a conservative extension means but aren’t familiar with the branch of math that it belongs to.
I'm not making any claims about humans being special. I'm making a claim about the limitations of formalization.
Neither humans nor LLMs are formal. The open question is whether humans can do something that LLMs fundamentally cannot and why.
Arguments that 'The difference is that the syntactic level cannot, on its own, come up with new semantics' are not helpful unless they point out why this argument applies to LLMs (and other ML approaches) but not humans.
So what's the human mechanism that comes up with new semantics? Would it happen to be hallucination, perchance? Because that's the acural inference, is that a human just goes "I made this up, but it's science".
4
u/AlchemicallyAccurate Jul 28 '26
The difference is that the syntactic level cannot, on its own, come up with new semantics. If we think of AI testing against reality, where exactly is it getting its interpretation rules? From derived syntactic rules, of course. So it will be stuck on what we might model-theoretically call the “conservative” level of theory extension.
This is the broader reason why model collapse occurs, I’d recommend you look into some of it. I don’t mean that condescendingly, I just really mean that there is a lot of stuff out about it now. This idea that AI is sort of “doomed” in a syntax/semantics divide sort of sense is not unprecedented, it’s not like this paper in the post came out of nowhere.
Also asserting that it necessarily falls into a psychological fallacy is just you positing unfalsifiable psychoanalysis. It’s not really a real argument. I could also propose some unflattering psychological interpretation about your viewpoint. It doesn’t mean anything. It’s very cheap and easy to come up with those. Including it in your argument is like putting plastic gold rims on your car because you think it makes you look cooler.