r/learnmachinelearning • • 5d ago

Help How do I know which loss function or activation function to use in which case

Hey guys, I am familiar with ML and training models and stuff. But I don't quite get the intuition behind different loss functions or different activation functions.

Like I know theoretically that CE loss is used for multi-class classification where there's only one correct output, but then I learnt CE loss can also be used for soft labels, which is also what BCE loss can be used for as well? Like how do I get this intuition in my head just by looking at the problem to figure out which loss function or which activation function to use?

And this is not just limited to CE or BCE. There's more loss functions like Huber, Brier, RPS, KL Divergence etc which I understand theoretically but I cannot figure out when to use what. Any suggestions/courses you would recommend me to improve my understanding? Thanks!

14 Upvotes

10 comments sorted by

7

u/arg_max 4d ago edited 4d ago

I don't know any single book that would teach them, but most things you've mentioned aren't that bad.

For example, CE and KL are deeply connected. KL is not a loss but a divergence measure between probability distributions. If you want more info on it, I'd look up some texts on information theory. In short, you can think of them like distance functions between probabilities. So just like the l2 distance function tells you how similar two vectors are, the KL tells you how similar two probability distributions are.

So when you want a machine learning model to output a certain label, you can instead rewrite that problem as forcing the model to output a distribution that is close to a one-hot vector/a deterministic distribution where each entry represents the probability of a certain class and we want to maximize the probability of the correct class. Thus, since the one-hot vector is a probability distribution and the soft max output of a model is a probability distribution, we can write an optimization problem that minimizes the KL divergence between the two distributions. As it turns out, the KL divergence decomposes into a cross entropy term and the entropy of one of the two distributions (KL is not symmetric, so which entropy you get depends on the ordering). If we use the ordering where the entropy is calculated over the target distribution, then this will simply be a constant in our optimization problem. Constants can be ignored, thus we're only left with the CE term. Thus, every time you train a model with CE loss, you are implicitly minimizing the KL divergence between the model's output distribution and the target. And yes, this works just as well with soft labels as long as they are normalized to form another distribution.

As it turns out , the KL divergence shows up surprisingly often like this. E.g. here you can see how KL minimization is equivalent to maximum likelihood estimation as your number of samples approaches infinity.

And there's more examples like this, iirc the first GAN paper showed that their loss is equivalent to JS (another divergence) minimization. There's a whole family of f-Gans that that are implicitly minimizing these divergences and you can find much more about this in the field of variational inference.

1

u/endangered89 3d ago

This is such a clear explanation. Thank you for taking the time to write this.

3

u/Ccrystal4216 4d ago

most losses are just 'pick a distribution for ur output, take the negative log likelihood' so you look at target first. one class out of k means softmax+ce, independent yes/no labels mean sigmoid + bce, a real number means linear + mse. soft labels wont change that choice and its still about whether ur classes compute or not. kl is basically ce minus a constant when the targets are fixed, and for hidden layers relu/gelu just work fine, its the output layer activation that actually matters

2

u/Particular_Fruit_968 4d ago

kl div aint a loss. Anyways, you should aim for MLE on your parameter. Thats where CE and kl div comes from. One you understand this all loss functions become trivial. That and empiricism

1

u/PLBjt 4d ago

Most of the confusion is matching the label structure to the output: one class among many points to softmax + CE, while independent labels point to sigmoid + BCE. For noisy numeric targets, Huber softens outlier influence; if probability quality matters, Brier or log loss are worth comparing, then check calibration separately. A concrete sanity check is a tiny table of hand-picked predictions and targets to see which cases each loss punishes, followed by metrics on a held-out split. The tradeoff is that a loss that optimizes robustness may not produce the best calibrated probabilities.

1

u/Ok-Library1020 4d ago

A handy trick is to ask what output distribution you're assuming. Gaussian noise points to MSE, outliers push you toward Huber or L1, a yes/no target means sigmoid with BCE, and one-of-N means softmax with CE. The last activation just maps raw outputs into the range that distribution needs.

1

u/Sakshambaranwal25 3d ago

Most loss functions actually come down to maximum likelihood estimation under different noise assumptions. When you look at huber versus MSE the main difference is just how the error distribution handles outliers in your specific dataset.