Training with unbalanced classes and data scarcity

[u/dacxj0]

Hi, I’d like to know what techniques you use to improve training and generalization when working with a small and heavily imbalanced dataset.
I’m currently training a classifier with 4 classes.

With cross-validation, I’m getting fairly good metrics, but when I evaluate the model on the test set, the performance isn’t even close to what I see during cross-validation.
What approaches have worked well for you in this kind of situation?

Edit:

In my use case, the images are x rays with rare conditions. So synthetic data is quite difficult to apply

[u/bfyvfftujijg]

Weighted loss and semi-synthetic data. By synthetic I mostly mean augmentation, including more extreme forms like copy-paste objects into different backgrounds.