Replay: Multi-modal Multi-view Acted Videos for Casual Holography

Roman Shapovalov, Yanir Kleiman, Ignacio Rocco, David Novotny, Andrea Vedaldi, Changan Chen, Filippos Kokkinos, Ben Graham, Natalia Neverova; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 20338-20348


We introduce Replay, a collection of multi-view, multi-modal videos of humans interacting socially. Each scene is filmed in high production quality, from different viewpoints with several static cameras, as well as wearable action cameras, and recorded with a large array of microphones at different positions in the room. Overall, the dataset contains over 3000 minutes of footage and over 5 million timestamped high-resolution frames annotated with camera poses and partially with foreground masks. The Replay dataset has many potential applications, such as novel-view synthesis, 3D reconstruction, novel-view acoustic synthesis, human body and face analysis, and training generative models. We provide a benchmark for training and evaluating novel-view synthesis, with two scenarios of different difficulty. Finally, we evaluate several baseline state-of-the-art methods on the new benchmark.

Related Material

[pdf] [supp] [arXiv]
@InProceedings{Shapovalov_2023_ICCV, author = {Shapovalov, Roman and Kleiman, Yanir and Rocco, Ignacio and Novotny, David and Vedaldi, Andrea and Chen, Changan and Kokkinos, Filippos and Graham, Ben and Neverova, Natalia}, title = {Replay: Multi-modal Multi-view Acted Videos for Casual Holography}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2023}, pages = {20338-20348} }