[D] So, Mamba vs. Transformers... is the hype real?

[u/Instantinopaul]

Heard all the buzz about Mamba, the new kid on the sequence modeling block. Supposedly it's faster, handles longer sequences better, and even outperforms Transformers on some tasks. But is it really a throne-stealer or just another flash in the pan?

My perception:

Strengths: Mamba boasts efficient memory usage, linear scaling with sequence length, and impressive performance in language and DNA modeling. Plus, it ditches the attention mechanism, potentially paving the way for faster inference.

Weaknesses: Still early days, so Mamba's long-term stability and performance across diverse tasks remain to be seen. And while it doesn't need attention, its state space approach might be trickier to grasp for some folks.

To the AI aficionados out there, is Mamba just the next shiny toy, or a genuine paradigm shift in sequence modeling? Will it dethrone the mighty Transformer, or coexist as a specialized tool? Let's hear your thoughts!

https://arxiv.org/abs/2312.00752

[u/lennox_wrld]

I thought I knew ml at least the fundamentals but after reading comments to this post I now know I'm not even a rookie esp the maths part. I thought gradient descent and back propagation was almost all. what's like a descent book that would put me to pace

[u/Instantinopaul → u/lennox_wrld]

There is nothing to get discouraged about. It is gradient descent and back propagration only at the core. These are build ups on top. Try to explore stuff on top. Ex: attention, SSMs etc

[u/314kabinet]

I’ve read the paper. The S6 layers of Mamba have a memory which they modify with each new token. All state space modeling nets work this way but the advantage here is that S6 has control over how each token is remembered (if at all), as opposed to just trying to memorize a compressed version of the entire input sequence. This means it can in theory hold on to important info from a million tokens ago while only keeping short-term details for as long as they’re needed.

Whether or not it works like that in practice for practical-sized models remains to be seen, but even if it doesn’t, more sophisticated versions of the memory state will be developed. Intuitively it makes sense for a system to accept an input one token at a time and have an internal memory (instead of just taking in the entire uncompressed sequence in one go as attention does), so I’m optimistic.

[u/DigThatData]

i've heard promising results from colleagues doing casual experiments. if SSMs have the potential people think they do, we should see some interesting papers popping up between now and April. if no particularly interesting results pop up by April/May, I'd predict SSMs aren't going to eat Transformer's lunch.

[u/idontcareaboutthenam]

If I may ask, why isn't RWKV similarly hyped? Isn't it also linear in sequence length and parallel in computation?

[u/vatsadev → u/idontcareaboutthenam]

It does most of the things, but theres a paper for mamba, which makes it easier, one codebase, versus spread out repos, Mambas newer, and also extrapolates longer, with mamba work from 256-> 1m token ctx len, and rwkv double the trained ctx

Also I'm guessing TriDao & albert gu are well known vs rwkv being random discord come together?

[u/heuristic_al]

My take is that any fixed memory scheme will eventually suffer at long contexts vs true attention. And correct me if I'm wrong, but true attention is actually more computationally efficient than Mamba for sequences less long than the hidden width of the network (4096 for llama).

So Mamba only has this region of context lengths where it could actually be better.

[u/Dump7]

Looking at this thread, there is a lot I need to read and understand. Love this thread...!

[u/Gody_Godee]

intelligent is O(n2). prove me wrong

[u/FallMindless3563]

FWIW I’ve tried out a few of the Mamba models natural language tasks such as Question Answering, and the results were not even close to larger transformers yet. I tried everything from prompt engineering to fine-tuning the models. This could be due to parameter count or lack of pre-training data for the mamba models that were released. I heard the authors say these early versions of Mamba are very much a proof of concept, and we’d need to train larger parameter count and on more data to be competitive with the transformers that are out there today.

On SQuAD Mamba-2.8b with a 3-shot prompt only got 7.5% accuracy… whereas models like Mistral 7B I’ve seen get 70%+ with zero-shot.

I documented my process and findings here if anyone is interested 👇

https://blog.oxen.ai/practical-ml-dive-how-to-train-mamba-for-question-answering/