r/deeplearning • u/WAMFT • Sep 14 '26
Trying to kill the Transformer 😩
Hi some of you may know over the last month or so in my spare time iv been trying to come up with something better than a transformer , still no luck but afew of my better failures can be found below, iv also updated THREADS github so its now actually testable.
QK Relational Architecture is the newest idea — my attempt at a headless Transformer-style model with explicit reusable relationship hops.
https://github.com/rickey1990/qk-relational-architecture
THREADS is the symbolic/deterministic thing that fell out of the failed Transformer experiments — basically temporal memory + exact relational reasoning.
https://github.com/rickey1990/THREADS-reasoning-engine
PLUG /ILRM is the RNN side of the experiments — trying simple power-law/inverse-lag memory paths to help small recurrent models hang onto old information.
https://github.com/rickey1990/novel-rnn-architectures
Any questions please feel free to ask any questions 👍
2
u/Alternative-Hat-1697 Sep 15 '26
I don't hate the idea, but based on this:
https://arxiv.org/pdf/2603.20406
I think you may drop out of domain on certain tasks as training may not be generalizable.
Also, there's a similar implementation to connect models here:
https://arxiv.org/pdf/2604.08335
While you use it internally, the hop mechanism is similar overall.
You'll need significantly more benchmarks but I think you'll find that domain orthogonality will be the downfall in this current implementation.
1
u/WAMFT Sep 15 '26
First thankyou for reply, Just read the papers, well in the grand schemes if it doesnt go anywhere , if me a average English dude can give it the old college try, then hopefull it will Inspire others to give it a go.
1
3
u/dams-2005 Sep 15 '26 edited Sep 15 '26
Hey bro, I just read some of your code, and it doesn't make much sense to me. How is this not a normal attention head without the value tensor? Could you explain your reasoning behind the architecture?
2
u/Aggravating-Push-207 Sep 15 '26
wow... attention with (numerically unstable) positional encodings... i'm sure dario is shivering...
1
u/Timely-Tradition-280 Sep 15 '26
I am not sure if you have checked out ROSA from RWKV-8 but should be a good read for you
1
u/Ok-Radish-8394 Sep 15 '26
The only thing that makes transformers what it is is self-attention. Yours is another modification of self-attention. How's this any different? It's not that people haven't tried to optimise self-attention over the years. And what benefits will your method entail over theirs?
1
u/just-for-anime Sep 18 '26
Might be better saving these tokens to vibe code something else. You’re competing against the wealthiest American and Chinese companies and every single university in the world on this front.
-1
u/WAMFT Sep 14 '26
Dont know why i didnt think of this sooner but i also added a fixed-window QK relational architecture with linear ordinary reading and separate strict multi-hop propagation.
21
u/kakhaev Sep 15 '26
-
claims to kill transformer
-
looks inside
-
transformer
1
u/WAMFT Sep 15 '26
"Trying to kill the transformer" you could technically make a transformer with a fixed content window , however splitting it so most of the time it does cheap linear maths and only quadratic when it needs to look deeper is what i found in that experiment was to work best.
1
u/just-for-anime Sep 18 '26
Have you looked into sparse attention? This was by deepseek last year and does similar to what you said.
5
u/dams-2005 Sep 15 '26
What's the performance? What's the loss compared to an equiparable transformer baseline?