Robust Single-view 3D Human Digitization via Explicit Geometric Field with Semantic Guidance

Ruizhi Liu, Paolo Remagnino; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3362-3378

We propose a novel single-view human body 3D reconstruction framework guided by the semantic field. We argue that full visibility of 2D human shape and the alignment between geometric and semantic features are vital for a robust 3D human reconstruction. Due to the single view setting, existing methods fail to predict robust and completed 2D human shapes, which leads to vulnerability against invisibility. The usage of a parametric model could inform the model with a human shape, but the shape prior is too general, resulting in converging to a general body shape, and poor generalization to different types of garments. In response to the aforementioned challenges, we propose a novel framework to reconstruct a photo-realistic 3D human mesh by estimating the overall 2D human shape from the given view via a Human shape predictor and predicting view-dependent explicit geometric field with an estimated semantic field via a novel Semantic-aware explicit geometric field. Then we ensure the queriablity of our view-dependent explicit geometric field via Semantic-aware Geometric Field Integration. Finally, we first propose a cascaded query strategy, termed Semantic-Guided Query Strategy to further combine the 2D and 3D features. Through extensive experiments, our framework surpasses all selected SoTA by a considerable gap in Chamfer and P2S distances on Thuman2.0, CAPE-NFP, and CAPE-FP datasets.