MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views

Runfa Li, Upal Mahbub, Vasudev Bhaskaran, Truong Nguyen; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 656-666

Current monocular 3D scene reconstruction (3DR) works are either fully-supervised or not generalizable or implicit in 3D representation. We propose a novel framework - MonoSelfRecon that for the first time achieves explicit 3D mesh reconstruction for generalizable indoor scenes with monocular RGB views by purely self-supervision on voxel-SDF (signed distance function). MonoSelfRecon follows an Autoencoder-based architecture decodes voxel-SDF and a generalizable Neural Radiance Field (NeRF) which is used to guide voxel-SDF in self-supervision. We propose novel self-supervised losses which not only support pure self-supervision but can be used together with supervised signals to further boost supervised training. Our experiments show that "MonoSelfRecon" trained in pure self-supervision outperforms current best self-supervised indoor depth estimation models and is comparable to 3DR models trained in fully supervision with depth annotations. MonoSelfRecon is not restricted by specific model design which can be used to any models with voxel-SDF for purely self-supervised manner.