Painting 3D Nature in 2D: View Synthesis of Natural Scenes From a Single Semantic Mask

Shangzhan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin, Kaicheng Yu, Yiyi Liao, Xiaowei Zhou; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 8518-8528

Abstract


We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which are inapplicable to natural scenes. Our key idea to solve this challenge is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translated to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic and multi-view consistent videos of a variety of natural scenes. The project website is https://zju3dv.github.io/paintingnature/.

Related Material


[pdf] [arXiv]
[bibtex]
@InProceedings{Zhang_2023_CVPR, author = {Zhang, Shangzhan and Peng, Sida and Chen, Tianrun and Mou, Linzhan and Lin, Haotong and Yu, Kaicheng and Liao, Yiyi and Zhou, Xiaowei}, title = {Painting 3D Nature in 2D: View Synthesis of Natural Scenes From a Single Semantic Mask}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2023}, pages = {8518-8528} }