DepthSegNet24: A Label-Free Model for Robust Day-Night Depth and Semantics

Phan Thi Huyen Thanh, The Hiep Nguyen, Minh Huy Vu Nguyen, Trung Thai Tran, Tran Vu Pham, Duc Dung Nguyen, Truong Vinh Truong Duy, Natori Naotake; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 2716-2733

This paper presents a novel multi-task model combining self-supervised monocular depth estimation and knowledge-distilled semantic segmentation that can perform both tasks simultaneously and consistently in both daytime and nighttime conditions. By leveraging the joint self-supervised and supervised knowledge distillation learning, the model can learn consistent and complementary representations of the two tasks to improve the generalization ability without relying on annotated ground-truth data. To address the extremely varying lighting conditions between day and night, we first synthesize night and day images from their corresponding real day and night images, and then train the model with the day-night image pairs to provide explicit correspondences between the two lighting conditions for capturing the contextual and detailed information in both scenarios. We also augment the model with a light enhancement module and a daytime depth pseudo-labels for achieving more accurate and robust depth and segmentation. Experimental results on Oxford RobotCar and nuScenes demonstrate the robustness of our model in diverse challenging lighting conditions.