Joint Motion Detection in Neural Videos Training

Niloufar Pourian, Alexey Supikov; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5693-5700

Neural radiance fields (NeRF) can produce photo realistic free-viewpoint images. Recently incremental neural video training approaches took a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Motion detection in neural videos via a frame-byframe approach can provide valuable cues to enable temporally stable neural videos suitable for interactive streaming. In addition motion cues can be used to guide the ray sampling phase to model dynamic regions more efficiently. Hence motion detection can be a key component in telepresence/social networking and immersive cloud gaming applications. In this paper we propose a novel approach that computes static/dynamic separation masks with high accuracy and spatial coherency across different views together with NeRF optimization process. This is enabled by using explicit deformation network instead of implicit motions/structure layers (novel network architecture) as well as novel specifically designed training schedule. To the best of our knowledge this is the first work that enables motion estimation via a frame-by-frame approach in a neural video training. The proposed work is desirable as it does not require buffer chunks of frames available before processing and hence is suitable for interactive streaming scenarios. Experimental results shows the effectiveness of the proposed motion detection approach in neural videos.