VIPNet: Combining Viewpoint Information and Shape Priors for Instant Multi-View 3D Reconstruction

Weining Ye, Zhixuan Li, Tingting Jiang; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3379-3395

While the multi-view 3D reconstruction task has made significant progress, existing methods simply fuse multi-view image features without effectively leveraging available auxiliary information, especially the viewpoint information for guiding and associating features of different views. To this end, we propose to enhance multi-view 3D reconstruction with the power of viewpoint information. Specifically, a simple-yet-effective viewpoint estimator is designed to learn and provide comprehensive viewpoint knowledge for locating and associating learned features from different views. Moreover, to improve the 3D reconstruction quality when 2D images of only very few viewpoints are available, we propose to learn the shape prior knowledge to provide sufficient shape information for compensating the limited 2D observations. Overall, we present VIPNet, benefiting from Viewpoint Information and Shape Prior learning for high-quality multi-view 3D reconstruction. Extensive experiments validate the effectiveness of the proposed VIPNet, which achieves state-of-the-art performance on challenging datasets and shows well generalization ability in real-world scenarios.