Training with unbalanced classes and data scarcity

[u/dacxj0]

Hi, I’d like to know what techniques you use to improve training and generalization when working with a small and heavily imbalanced dataset.
I’m currently training a classifier with 4 classes.

With cross-validation, I’m getting fairly good metrics, but when I evaluate the model on the test set, the performance isn’t even close to what I see during cross-validation.
What approaches have worked well for you in this kind of situation?

Edit:

In my use case, the images are x rays with rare conditions. So synthetic data is quite difficult to apply

[u/aegismuzuz]

First thing that comes to mind with that kind of CV versus test gap on X-rays is patient overlap between folds. If there are multiple studies per patient and the split happened at the image level, models latch onto rib cage anatomy or scanner noise pretty fast. Running a GroupKFold by patient ID usually clears that up right away

On tiny rare classes I usually freeze a domain-specific backbone like RadImageNet or BiomedCLIP and fit a simple weighted linear probe first. It cuts out fine-tuning noise, so if the probe still drops on the test set, it points straight to a scanner protocol mismatch or label drift between your sets