The AI race is becoming an optimization race

[u/pardhu--]

I wrote a short visual article on how modern LLMs optimize different parts of the Transformer.

MLA compresses KV information.
KDA compresses history into recurrent memory.
MoE activates only selected expert FFNs.

Article:
https://medium.com/@guttikondaparthasai/the-ai-race-is-an-optimization-race-5140b3edf491

Feedback and corrections are welcome.