ProMM-RS: Exploring Probabilistic learning for Multi-Modal Remote Sensing Image Representations

Nicolas Houdré, Diego Marcos, Dino Ienco, Laurent Wendling, Camille Kurtz, Sylvain Lobry; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 554-562

Remote sensing imagery offers diverse modalities such as synthetic aperture radar and multispectral data which can bring rich complementary and valuable information about observed scenes. This information is of paramount importance for downstream applications (e.g. land cover mapping natural resources monitoring human settlement characterization) that may benefit from such complementarity. Remote sensing imagery often suffers from a lack of labeled data which can hamper the learning of good representations via state-of-the-art supervised methods. Self-supervised learning has thus emerged as a promising paradigm for remote sensing feature extraction enabling the extraction of meaningful features without reliance on labeled data. While existing multi-modal contrastive models effectively capture shared information between modalities they often struggle to account for the inherent heterogeneity of multi-modal remote sensing data. This limitation prevents them from fully leveraging the complementarity of multi-modal remote sensing data. Probabilistic representation learning has emerged as a powerful approach to capture the inherent uncertainty and diversity in multi-modal relationships. In this paper we present ProMM-RS a novel multi-modal self-supervised training framework incorporating a joint probabilistic embedding space to explicitly model the uncertainty of representations between different inputs and modalities. We evaluate our learned representations with a scene classification downstream task from Sentinel optical and radar images effectively showing the potential of probabilistic embeddings as a way to measure the relevancy of each modality representation especially under an obstructed dataset.