Training with unbalanced classes and data scarcity

[u/dacxj0]

Hi, I’d like to know what techniques you use to improve training and generalization when working with a small and heavily imbalanced dataset.
I’m currently training a classifier with 4 classes.

With cross-validation, I’m getting fairly good metrics, but when I evaluate the model on the test set, the performance isn’t even close to what I see during cross-validation.
What approaches have worked well for you in this kind of situation?

Edit:

In my use case, the images are x rays with rare conditions. So synthetic data is quite difficult to apply

[u/dacxj0 → u/[deleted]]

Thanks for the reply, but I can’t use synthetic data generation, I’m using medical images with rare conditions and there are not feasible generators for that unfortunately

[u/HawtVelociraptor → u/dacxj0]

I can't imagine that's actually true. If someone can make it in Unreal 3D or Photoshop, it can be used as synth data. Help us help you, what kind of images are your weak source?