SACReg: Scene-Agnostic Coordinate Regression for Visual Localization

Jerome Revaud, Yohann Cabon, Romain Brégier, Jongmin Lee, Philippe Weinzaepfel; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 688-698

Scene coordinates regression (SCR) i.e. predicting 3D coordinates for every pixel of a given image has recently shown promising potential. However existing methods remain limited to small scenes memorized during training and thus hardly scale to realistic datasets and scenarios. In this paper we propose a generalized SCR model trained once to be deployed in new test scenes regardless of their scale without any finetuning. Instead of encoding the scene coordinates into the network weights our model takes as input a database image with some sparse 2D pixel to 3D coordinate annotations extracted from e.g. off-the-shelf Structure-from-Motion or RGB-D data and a query image for which are predicted a dense 3D coordinate map and its confidence based on cross-attention. At test time we rely on existing off-the-shelf image retrieval systems and fuse the predictions from a shortlist of relevant database images w.r.t. the query. Afterwards camera pose is obtained using standard Perspective-n-Point (PnP). Starting from selfsupervised CroCo pretrained weights we train our model on diverse datasets to ensure generalizabilty across various scenarios and significantly outperform other scene regression approaches including scene-specific models on multiple visual localization benchmarks. Finally we show that the database representation of images and their 2D-3D annotations can be highly compressed with negligible loss of localization performance.