-
[pdf]
[supp]
[bibtex]@InProceedings{Hong_2024_CVPR, author = {Hong, Sunghwan and Jung, Jaewoo and Shin, Heeseong and Yang, Jiaolong and Kim, Seungryong and Luo, Chong}, title = {Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {20196-20206} }
Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis
Abstract
This work delves into the task of pose-free novel view synthesis from stereo pairs a challenging and pioneering task in 3D vision. Our innovative framework unlike any before seamlessly integrates 2D correspondence matching camera pose estimation and NeRF rendering fostering a synergistic enhancement of these tasks. We achieve this through designing an architecture that utilizes a shared representation which serves as a foundation for enhanced 3D geometry understanding. Capitalizing on the inherent interplay between the tasks our unified framework is trained end-to-end with the proposed training strategy to improve overall model accuracy. Through extensive evaluations across diverse indoor and outdoor scenes from two real-world datasets we demonstrate that our approach achieves substantial improvement over previous methodologies especially in scenarios characterized by extreme viewpoint changes and the absence of accurate camera poses.
Related Material