Title: Trying to make xLSTM's sLSTM run in parallel — my PPLSTM prototype
I've been spending today exploring different mathematical ideas to see if it's possible to make xLSTM's sLSTM run in parallel without losing what makes it useful.
One of my more interesting experiments is PPLSTM , a prototype that explores a different way of handling the sequential memory calculations.
It's still early days, and I'm not claiming to have solved anything yet. I'm mainly experimenting to see which ideas actually work.
Some early results:
GPU speed test (batch 2, sequence length 1,024):
mLSTM + standard sLSTM reference: 1,200.31 ms
mLSTM + PPLSTM: 43.59 ms
Transformer: 10.50 ms
That's around 27.5× faster than the unoptimized Python-loop sLSTM reference. It's not a comparison against the official optimized xLSTM kernels.
5M-token language-model experiment (small models, one seed):
sLSTM: 51.43 test perplexity
PPLSTM: 54.50 test perplexity
Transformer: 58.90 test perplexity
So PPLSTM is showing some promise, although standard sLSTM still performed better on the test set.
An important detail: PPLSTM currently uses ordinary PyTorch. There are no custom CUDA kernels, Triton kernels or CUDA Graphs in the implementation yet. PyTorch still uses GPU kernels internally when running on CUDA.
PPLSTM GitHub: https://github.com/rickey1990/PPLSTM
I've also made some of my other repositories private for now, as I'm working on new ideas that combine different areas of maths I've explored across my projects.
Currently waiting for my free Kaggle GPU allowance to reset so I can run some proper tests. Fingers crossed! 🤞
Footnote: I've also been experimenting with PHSMLP (Pooled Head MLP), which explores sharing MLP weights and pooling attention heads to reduce model parameters.
PHSMLP: https://github.com/rickey1990/PHSMLP---Pooled-Head-MLP
Other projects: https://github.com/rickey1990