Towards Dense 3D Reconstruction for Mixed Reality in Healthcare: Classical Multi-View Stereo vs Deep Learning

Kristina Prokopetc, Romain Dupont; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 0-0

Faithfully reproducing surroundings in 3D is a key-component in Mixed Reality for medical training in neonatology, where a user sees a hospital room in a Virtual Reality helmet while retaining tangible interaction with a baby mannequin and various medical tools. Deep learning solutions have high claims against classical methods but their performance in real-life application remains unclear. To fill this blank, we present a comparative study of depth map based Multi-View Stereo methods for dense 3D reconstruction. We compare classical state-of-the-art methods to their learned counterparts and assess their robustness to weakly-textured and reflective surfaces as well as accuracy on thin structures both globally and locally. We also analyze the effect of depth filtering along with computational effort. Our experiments reveal various factors which contribute to the performance gap between the methods that we discuss in detail. This study is the first to evaluate traditional dense geometry reconstruction methods against brand-new deep learning models. It helps to better understand what suits best the challenges of hospital environments. Furthermore, it builds a solid analytic ground to underscore the strengths and weaknesses of the learned methods.