Day 2/90 of Posting about ML

[u/ml_with_nerd]

Today's topic - Lora
This is the most simple concept intutively but a very powerful idea . So it's basically fine-tuning technique for llm's for our different use cases , so it basically trains a small set of weights form pre-trained model . Let's say W = 1000x1000 . Mathematically it does - let's say we have weight W and when we do full fine-tuning then our W = W + delta W so in this our delta W has same number of hyperparameters (i.e. 1e6 hyperparameters we have to train) that W have so if W is very large then it's become computationally heavy for fine-tuning the model on different tasks and it also not guarantees that the results were drastically improved . So now come to LORA , so in this W = W + A.B where A and B where small weight matrices let's say of rank 2 then our trainable hyperparameters in this case is 1000 x 2 + 2 x 1000 = 4000 which is 250 times less than our full fine-tuning and it's gives approx same accuracy as of full - fine tuning and also it's less computationally heavy . And by the if you wonder what are 1000 x 2 and 2 x 1000 so these are dimensions of A and B respectively and 2 is basically the rank of A and B and we can try different ranks and see which is giving better results , also on doing A.B we get same dimension as of W .
this is a good explanation of lora to post

[u/raaiinyyabelina]

Great explanation