Uniting Stereo and Depth-From-Defocus: A Thin Lens-Based Variational Framework for Multiview Reconstruction

Robert D. Friedlander, Huizong Yang, Anthony J. Yezzi; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021, pp. 4406-4415

The problem of reconstructing three-dimensional (3D) scene geometry and radiometry from images is an important problem in computer vision and has applications in a variety of fields such as medicine and artifact preservation. However, state-of-the-art multiview algorithms assume a pinhole camera that incorrectly models defocus blur as a property of the scene instead of a property of the imaging process. We address the problem of dense 3D shape reconstruction from multiple viewpoints in situations where the image data exhibits noticeable defocus. We develop a mathematical framework for a fully generative variational algorithm that iteratively deforms an estimate of the foreground surface shapes and scene radiance such that irradiance estimates given by the thin lens forward model are photometrically consistent with the actual image data. This framework is founded on novel geometric computations of flux differentials across an evolving surface as well as gradients along occluding boundaries and their projections. While more work is needed to make them fit for practical use, the future potential of methods based on these computations is shown with experiments reconstructing simple object shapes from both synthetically generated and real defocused images. While our reconstruction algorithm has a higher computational cost than pinhole-based methods due to the more general optical model, it better reconstructs object proportions as well as sharp features that are blurred due to image defocus. As such, our geometry-based method provides a unified framework that extends the applicability of multiview reconstruction techniques to the poorly supported domain of defocused images where state-of-the-art pinhole-based methods fail.