Modality Weighting in Multimodal Variational Autoencoders - Yegin, Melike Nur; Amasyali, Mehmet Fatih (Innovations in Intelligent Systems and Applications Conference (Online), 2022)
Learning the same subject from different sources is known as multimodal learning in the machine learning literature. In this approach some of the modalities may be more reliable than others and we may want to learn them more to a better understanding of the subject. In this study, we propose weighting the reliable modalities in multimodal learning process. We introduce both weighting with fixed coefficients and several methods for weighting automatically. We compare the proposed methods with classical methods on artificial Gaussian data sets and MNIST/FashionMNIST data sets and we have obtained significiant results.