Fast and Accurate: Video Enhancement Using Sparse Depth

Yu Feng, Patrick Hansen, Paul N. Whatmough, Guoyu Lu, Yuhao Zhu; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 4492-4500

Abstract


This paper presents a general framework to build fast and accurate algorithms for video enhancement tasks such as super-resolution, deblurring, and denoising. Essential to our framework is the realization that the accuracy, rather than the density, of pixel flows is what is required for high-quality video enhancement. Most of prior works take the opposite approach: they estimate dense (per-pixel)--but generally less robust--flows, mostly using computationally costly algorithms. Instead, we propose a lightweight flow estimation algorithm; it fuses the sparse point cloud data and (even sparser and less reliable) IMU data available in modern autonomous agents to estimate the flow information. Building on top of the flow estimation, we demonstrate a general framework that integrates the flows in a plug-and-play fashion with different task-specific layers. Algorithms built in our framework achieve 1.78x -- 187.41x speedup while providing a 0.42dB - 6.70 dB quality improvement over competing methods.

Related Material


[pdf] [arXiv]
[bibtex]
@InProceedings{Feng_2023_WACV, author = {Feng, Yu and Hansen, Patrick and Whatmough, Paul N. and Lu, Guoyu and Zhu, Yuhao}, title = {Fast and Accurate: Video Enhancement Using Sparse Depth}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2023}, pages = {4492-4500} }