Modeling Detailed Human Geometry with Adaptive Local Refinement

Bang Du, Kunyao Chen, Haochen Zhang, Fei Yin, Baichuan Wu, Truong Nguyen; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5620-5630

Estimating clothed human body shapes from monocular images has been a difficult problem due to occlusions varying poses and diverse clothing styles. Current methods involve directly regressing for either 3D positions of primitives or values in a volumetric space but they struggle to balance generalization and accuracy leading to suboptimal results. In this paper we introduce a novel two-step framework that efficiently combines 2D and 3D representations to achieve both accurate surface detail inference and strong generalization capabilities: addressing challenging poses by occlusions and varying clothing styles. Our approach first uses an image-to-image translation framework to estimate a rough shape which serves as an initial approximation of the human body. This step effectively captures global structure and coarse details while being computationally efficient. Next we employ a dedicated refinement module to enhance the surface details for a high-fidelity result. It utilizes an attention-based strategy that allows the 3D refinement module to focus on regions of interest such as areas with complex clothing or occlusions. This strategy effectively improves the overall quality of the inferred shape by generating high-density patches of points in challenging regions. Our experiments show that with the attention-based strategy the proposed method outperforms state-of-the-art methods in terms of both qualitative and quantitative measures demonstrating its effectiveness in handling diverse clothing styles and poses.