Narrowing the Synthetic-to-Real Gap for Thermal Infrared Semantic Image Segmentation Using Diffusion-based Conditional Image Synthesis

Christian Mayr, Christian Kubler, Norbert Haala, Michael Teutsch; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 3131-3141

Semantic segmentation is the task of assigning a semantic class to each pixel in an image. Due to the high annotation efforts for fully supervised learning of Deep Neural Networks (DNNs) for this task only rather few comprehensive public datasets exist. This is particularly the case for thermal infrared imagery. To overcome this lack of training data we propose to utilize conditional image synthesis in the thermal infrared spectrum. Existing semantic segmentation maps are used to condition the image generation process using pretrained text-to-image diffusion models. Therefore we use the recently published ControlNet and retrain it to synthesize thermal infrared images for given semantic maps. In this way we can generate large numbers of synthetic images that we can directly use together with the related segmentation map to train reference semantic segmentation approaches in the thermal infrared spectrum. Our experiments demonstrate that we achieve near state-of-the-art performance with pure synthetic training data on the recently published Full-time Multi-modality Benchmark (FMB) dataset and that our trained model shows better generalization ability across datasets. We provide code at https://github.com/HensoldtOptronicsCV/TIRControlNet.