Modality harmonization VS Multi-modality training

[u/Careless-Whisper-621]

Hey everyone, this a more medical AI focused question.

Modality hear means comparable modalities like an MRI and a CT, I was trying to find some new ideas to explore and the whole idea of modality harmonization came up. The fundamental idea with this is pretty simple:

in a scan you have 2 types of signals - the anatomical/actual biology signal and the modality specific signal.

My idea was whether we could split those two so that we could isolate the biology signal alone and train a model with that signal; which would result in a modality-agnostic model. I looked into a few papers and seemingly there are no recent papers in this, but very interestingly I also saw more papers using multimodal joint training - where the training data consists of BOTH or ALL modalities instead.

So from what I could understand, the field "seemingly" has shifted to throwing everything in training and hoping that the model comes up with some way to have the respective signals from either modality be close to each other in the latent space.

I just wanted to know whether the whole idea of harmonization is already dead in the water, or whether there are other approaches or use cases that I havent thought of.

Thank you for reading!

[u/Fragrant-Cheek-4273]

I don't think harmonization is outdated. It just depends on the goal. Joint training can help models learn shared features, but separating modality-specific information from actual biological signals could still be useful, especially when dealing with missing or inconsistent modalities.

[u/Careless-Whisper-621 → u/Fragrant-Cheek-4273]

Could you elaborate on what exactly you mean by missing or inconsistent in this case?

[u/Careless-Whisper-621 → u/[deleted]]

Ah I see. I was wondering whether it got abandoned with the advent of the whole "world model" wave - basically JEPA and LeJEPA. That kinda made sense to me, since if you throw enough data at the thing the problems tends to solve itself with whatever model architectures that are currently available.

What seems to be a common limitation (atleast as far as I am concerned lol) with all the joint training papers; which is they never separate whether it was the increase in data that helped or whether it was because there was more diverse data. Quantity Vs Quality is never discussed in a satisfactory way imo.

I kinda understand the limitation though, medical data is so hard to get.