Multi-View Stereo with Single-View Semantic Mesh Refinement

Andrea Romanoni, Marco Ciccone, Francesco Visin, Matteo Matteucci; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 706-715


Semantic 3D reconstruction only recently witnessed an increasing share of attention from the Computer Vision community. Semantic annotations allow to enforce class-dependent priors, which can improve 3D reconstruction. Existing methods propose volumetric approaches; even if successful, they do not scale well. In this paper we propose a novel method to refine both the geometry and the semantic labeling of a given mesh. We refine the geometry through a variational method that optimizes a composite energy made of a state-of-the-art pairwise photo-metric term and a novel single-view term that models the semantic consistency between the labels of the 3D mesh and those of the segmented images. We update the semantic labeling through a novel Markov Random Field that, together with the usual data and smoothness terms, takes into account class-specific priors estimated directly from the annotated mesh, in contrast to state-of-the-art methods that are based on handcrafted or learned priors.

Related Material

[pdf] [arXiv]
author = {Romanoni, Andrea and Ciccone, Marco and Visin, Francesco and Matteucci, Matteo},
title = {Multi-View Stereo with Single-View Semantic Mesh Refinement},
booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops},
month = {Oct},
year = {2017}