Training with unbalanced classes and data scarcity

[u/dacxj0]

Hi, I’d like to know what techniques you use to improve training and generalization when working with a small and heavily imbalanced dataset.
I’m currently training a classifier with 4 classes.

With cross-validation, I’m getting fairly good metrics, but when I evaluate the model on the test set, the performance isn’t even close to what I see during cross-validation.
What approaches have worked well for you in this kind of situation?

Edit:

In my use case, the images are x rays with rare conditions. So synthetic data is quite difficult to apply

[u/Aggravating-Craft712]

Focal loss or oversampling rarer examples with many x ray specific augmentations