Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training

Runze He, Shaofei Huang, Xuecheng Nie, Tianrui Hui, Luoqi Liu, Jiao Dai, Jizhong Han, Guanbin Li, Si Liu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 6966-6975

Abstract

In this paper we target the adaptive source driven 3D scene editing task by proposing a CustomNeRF model that unifies a text description or a reference image as the editing prompt. However obtaining desired editing results conformed with the editing prompt is nontrivial since there exist two significant challenges including accurate editing of only foreground regions and multi-view consistency given a single-view reference image. To tackle the first challenge we propose a Local-Global Iterative Editing (LGIE) training scheme that alternates between foreground region editing and full-image editing aimed at foreground-only manipulation while preserving the background. For the second challenge we also design a class-guided regularization that exploits class priors within the generation model to alleviate the inconsistency problem among different views in image-driven editing. Extensive experiments show that our CustomNeRF produces precise editing results under various real scenes for both text- and image-driven settings. The code is available at: https://github. com/hrz2000/CustomNeRF.

Related Material

[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{He_2024_CVPR, author = {He, Runze and Huang, Shaofei and Nie, Xuecheng and Hui, Tianrui and Liu, Luoqi and Dai, Jiao and Han, Jizhong and Li, Guanbin and Liu, Si}, title = {Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {6966-6975} }