Ground Truth Accuracy and Performance of the Matching Pipeline

Josef Maier, Martin Humenberger, Oliver Zendel, Markus Vincze; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 29-39

Feature matching quality strongly influences the accuracy of most computer vision tasks. This led to impressive advances in keypoint detection, descriptor calculation, and feature matching itself. To compare different approaches and evaluate their quality, datasets from related tasks are used. Unfortunately, none of these datasets actually provide ground truth (GT) feature matches. Thus, matches can only be approximated due to repeatability errors of keypoint detectors and inaccuracies of GT. In this paper, we introduce ground truth matches (GTM) for several well known datasets. Based on the provided spacial ground truth, we automatically generate them using popular feature types. Currently, feature matching evaluation is typically performed using precision and recall. The introduced GTM additionally enable evaluation with accuracy and fall-out. The datasets were manually annotated, on the one hand to evaluate the precision and unambiguousness of the GTM, and on the other hand to determine the accuracy of the ground truth provided with the datasets. Using GTM, we present an evaluation of multiple state-of-the-art keypoint-descriptor combinations as well as matching algorithms.