-
[pdf]
[supp]
[arXiv]
[bibtex]@InProceedings{Du_2021_ICCV, author = {Du, Yilun and Zhang, Yinan and Yu, Hong-Xing and Tenenbaum, Joshua B. and Wu, Jiajun}, title = {Neural Radiance Flow for 4D View Synthesis and Video Processing}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {14324-14334} }
Neural Radiance Flow for 4D View Synthesis and Video Processing
Abstract
We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene. By enforcing consistency across different modalities, our representation enables multi-view rendering in diverse dynamic scenes, including water pouring, robotic interaction, and real images, outperforming state-of-the-art methods for spatial-temporal view synthesis. Our approach works even when being provided only a single monocular real video. We further demonstrate that the learned representation can serve as an implicit scene prior, enabling video processing tasks such as image super-resolution and de-noising without any additional supervision.
Related Material