HMGS: Hybrid Model of Gaussian Splatting for Enhancing 3D Reconstruction with Reflections

Hengbin Zhang, Chengliang Wang, Ji Liu, Tian Jiang, Yonggang Luo, Lecheng Xie; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 3819-3835

The advent of 3D Gaussian Splatting (3D-GS) marks a significant breakthrough in the field of 3D reconstruction, leveraging GPU rasterization technology to achieve real-time rendering with state-of-the-art quality. However, 3D-GS is limited by the capacity of low-order spherical harmonics to represent high-frequency reflective attributes, often resulting in the loss of critical information in scenes with highlights and reflections. To address this limitation, we propose HMGS, a hybrid model that enhances the original 3D-GS's ability to capture reflective colors. Our approach employs a neural network to learn color components from both the camera viewing direction and the reflected light direction, which are then jointly trained with the original 3D-GS model. Furthermore, we introduce a smoothing loss for the reflective color component, effectively decoupling the two color components. Our method significantly improves the reconstruction performance of 3D-GS on datasets featuring metallic sheen, light reflections, and shadows, while also enhancing reconstruction quality on general datasets.