pixelNeRF: Neural Radiance Fields From One or Few Images

Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 4578-4587

Abstract


We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields (NeRFs) involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, allowing it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks under category specific and category agnostic settings. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes as well as real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Yu_2021_CVPR, author = {Yu, Alex and Ye, Vickie and Tancik, Matthew and Kanazawa, Angjoo}, title = {pixelNeRF: Neural Radiance Fields From One or Few Images}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2021}, pages = {4578-4587} }