SCOTCH and SODA: A Transformer Video Shadow Detection Framework

Lihao Liu, Jean Prost, Lei Zhu, Nicolas Papadakis, Pietro Liò, Carola-Bibiane Schönlieb, Angelica I. Aviles-Rivero; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10449-10458

Abstract

Shadows in videos are difficult to detect because of the large shadow deformation between frames. In this work, we argue that accounting for shadow deformation is essential when designing a video shadow detection method. To this end, we introduce the shadow deformation attention trajectory (SODA), a new type of video self-attention module, specially designed to handle the large shadow deformations in videos. Moreover, we present a new shadow contrastive learning mechanism (SCOTCH) which aims at guiding the network to learn a unified shadow representation from massive positive shadow pairs across different videos. We demonstrate empirically the effectiveness of our two contributions in an ablation study. Furthermore, we show that SCOTCH and SODA significantly outperforms existing techniques for video shadow detection. Code is available at the project page: https://lihaoliu-cambridge.github.io/scotch_and_soda/

Related Material

[pdf] [arXiv]
[bibtex]
@InProceedings{Liu_2023_CVPR, author = {Liu, Lihao and Prost, Jean and Zhu, Lei and Papadakis, Nicolas and Li\`o, Pietro and Sch\"onlieb, Carola-Bibiane and Aviles-Rivero, Angelica I.}, title = {SCOTCH and SODA: A Transformer Video Shadow Detection Framework}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2023}, pages = {10449-10458} }