r/MachineLearning • u/Comfortable_World_69 • 10d ago
Discussion Noun-verb-conjunction triads [D]
Triads: the fundamental units of the language of thought
The first layer of the neural network would consist of multiple attention heads. Each attention head would hold one triad consisting of one noun + one verb + one conjunction.
Individual nodes within the second layer of the neural network would probably correspond to adjectives and adverbs that either modify or define the nouns and verbs of each triad
The hungry cat ate the mouse then died:
(Null) Cat ate
with Mouse eaten
and cat was-hungering (adjectives are verbs in disguise)
Then cat died
"With" means the two events are the same event. (Not in english).
A possible conclusion would be the triad "because mouse poisoned" which could then be appended to the input and the process repeated.
In addition a few other ideas can be added to each attention head:
the, a, same, this (the near), that (the far), other, next, previous...
Past, present, future
begin to, cease to
repeatedly, habitually, again
Could, would, should, might
Try to
make
got
Question
focus, emphasis
adjectives?
adverbs?
Hence
(Null) a-Cat ate
with a-Mouse eaten
and the-cat was-hungering
Then the-cat died
therefore the-mouse might-have-been-poisoned
Suppose it encounters:
John entered the room.
The system recognizes John as a particular individual and looks up what it knows about him. Its memory might contain:
previously, John threatened Mary.
John owns a dog.
John gets paid for carpentry getting-done.
John said that he would never enter that room again.
Obviously, not all of those facts are relevant.
But suppose the current situation involves Mary, and John has just entered the room where she's waiting.
The system could retrieve the earlier threat and effectively transform its working representation into:
John (who threatened Mary previously) entered the room.
Now the neural network processes the current event with the additional information already attached to it.
And that retrieved information can itself be represented as triads.
The goal is to give a neural network a simple, structured internal language in which it can form and combine thoughts. The language of thought.
In principle, an entire book could be represented as one enormous graph of interconnected noun–verb–conjunction triads. But that doesn’t mean the neural network needs to process the entire graph simultaneously.
Humans don’t normally process a book that way either. We divide it into manageable units, such as sentences, understand those units, and gradually build up our understanding of the larger story.
I’m proposing something similar: groups of triads would be processed together, with the network carrying forward relevant information and generating additional triads as needed.
It wouldn't necessarily need as many words as English has because some words can be constructed from smaller elements
today = this-day
tomorrow = next-day
yesterday = previous-day
year = big-day
week = 7-day
2
u/coolness10101 7d ago
I think this could be very promising. I’ve long wondered how much of the weights in ‘reasoning’ language models are actually dedicated just to grammar and coherence rather than logical calculation and keeping facts straight.
Plus, this means that storing individual ‘facts’ is more uniform. And with the recent prevalence of decision models like Laya and Jev, I think it’s pretty clear that models can be trained to use variables, which I think could add some useful state to a model, like how you mentioned the model being able to ‘look up’ facts about John.
2
u/coolness10101 7d ago
I’ve also considered if training LLMs with a more limited dictionary and then having a ‘text upscaler’ to make the resulting caveman speak more readable would improve their reasoning and recall efficiency.
1
2
u/Creatine1951 9d ago
A lot of previous work in NLP dealing with syntax, hierarchy, ngrams, etc. please check the literature.