Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval From a Single Image

Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, Angela Dai; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 12589-12599

Abstract


3D perception of object shapes from RGB image input is fundamental towards semantic scene understanding, grounding image-based perception in our spatially 3-dimensional real-world environments. To achieve a mapping between image views of objects and 3D shapes, we leverage CAD model priors from existing large-scale databases, and propose a novel approach towards constructing a joint embedding space between 2D images and 3D CAD models in a patch-wise fashion -- establishing correspondences between patches of an image view of an object and patches of CAD geometry. This enables part similarity reasoning for retrieving similar CADs to a new image view without exact matches in the database. Our patch embedding provides more robust CAD retrieval for shape estimation in our end-to-end estimation of CAD model shape and pose for detected objects in a single input image. Experiments on in-the-wild, complex imagery from ScanNet show that our approach is more robust than state of the art in real-world scenarios without any exact CAD matches.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Kuo_2021_ICCV, author = {Kuo, Weicheng and Angelova, Anelia and Lin, Tsung-Yi and Dai, Angela}, title = {Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval From a Single Image}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {12589-12599} }