Unifying Low-Resolution and High-Resolution Alignment by Event Cameras for Space-Time Video Super-Resolution

Hoonhee Cho, Jae-Young Kang, Taewoo Kim, Yuhwan Jeong, Kuk-Jin Yoon; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 9491-9502

Event cameras deliver asynchronous pixel intensity changes which result in sparse event data that offers the advantages of high temporal resolution. These high temporal characteristics make researchers naturally incorporate event cameras into video frame interpolation (VFI) and video super-resolution (VSR). In this paper we make the first attempt to solve the space-time video super-resolution (STVSR) task effectively addressing both VFI and VSR simultaneously by leveraging temporally dense events. STVSR aims to generate intermediate high-resolution (HR) videos between consecutive low-resolution (LR) frames. To fully exploit the high temporal frequency of events for STVSR we focus on temporal alignment in two stages at low-resolution and after up-sampling in high-resolution. In temporal alignment at low-resolution to upsample spatial dimensions effectively we leverage high temporal features to preserve spatial context. On the other hand for temporal alignment at the high-resolution stage we employ a deformable sampling process from events to achieve accurate alignment with forward and backward directions. In addition we provide the SuperREST dataset which features high-frequency details and complex motion in an RGB-Event setup. Experimental results on several datasets demonstrate that our method achieves a significant performance gain on STVSR tasks with low computational cost. Our codes and datasets are available at https://github.com/Chohoonhee/ESTNet.