Shadow Removal based on Diffusion Segmentation and Super-resolution Models

Chenghua Li, Bo Yang, Zhiqi Wu, Gao Chen, Yihan Yu, Shengxiao Zhou; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 6045-6054

Shadow removal is one of essential tasks among image restoration tasks which aims to eliminate the visual semantic information hidden or obscured by the shadow in the image to the largest extent. Variations in lighting and the diverse complexity of shadow depth and color resulting from random background factors are common in the shadow removal task. To address these challenges this paper proposes a novel interactive shadow removal architecture based on the diffusion model semantic segmentation and multimodal large language model. Our method utilizes a powerful diffusion model to generate shadow-free images with fewer artifacts and super-resolution models to enhance image details. A universal semantic segmentation model is also involved to reduce percpetual dissonance caused by slicing inference. Furthermore we integrate the capabilities of multimodal large language models to realize prior rule-based optimization. Leveraging the exceptional generative capability of diffusion model and elaborate cooperation among all the modules our method achieves outstanding perceptual performance on WSRD dataset. We conduct comprehensive experiments to demonstrate the effectiveness of our approach and share insights gained during the participation in the NTIRE 2024 Image Shadow Removal Challenge.