Latent Feature-Guided Diffusion Models for Shadow Removal

Kangfu Mei, Luis Figueroa, Zhe Lin, Zhihong Ding, Scott Cohen, Vishal M. Patel; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 4313-4322

Recovering textures beneath shadows has remained a challenging problem due to the inherent difficulty of inferring shadow-free scenes from shadow images. In this paper, we propose the use of diffusion models as they offer a promising approach to gradually refine details of shadow regions during the diffusion process. Our method improves the process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images, which has been a limitation of conventional methods that condition on degraded images only. Additionally, we propose to alleviate the potential local optimum of model optimization by fusing noise features with the diffusion network. We demonstrate the effectiveness of our approach, where it outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset and outperforms the previous best method by 82% in terms of RMSE on the DeSOBA dataset for instance-level shadow removal.