\section{Introduction}

% \RS{Why did you choose ORBIT for the dataset name? I don't think it's explained anywhere.} \ds{I added it at the end of the second last paragraph of intro.}

Recovering 3D geometry and ego-motion (i.e., camera pose) from videos is central to spatial reasoning, visual prediction, fine-grained robotic interaction, AR/VR, and learning controllable (3D and 4D) generative world models. 
A central component of this task is recovering structure-from-motion (SfM), where we take videos (or images) and produce camera poses and 3D scene geometry.

Current SfM methods work well on short videos featuring static scenes, but, even with significant recent progress, current methods can still produce errors or fail entirely on complex dynamic scenes---for instance, scenes featuring significant numbers of moving objects, regions of complex motion (e.g., waving foliage or moving water), and complex reflectance from specular surfaces; see \Cref{fig:teaser}.
Furthermore, it is difficult to gauge the performance of SfM methods on complex real-world videos, as benchmarks with ground truth camera poses that reflect these challenges are sorely missing.

Unfortunately, it is very difficult to obtain high-quality ground truth camera trajectories and 3D information for challenging real-world videos. 
Much of the prior work uses COLMAP~\cite{schonberger2016colmap} as a source of truth, whereas in reality, we observe that COLMAP fails on \textit{many} real-world videos.
%%%%%%%%%
% Current methods for SfM mainly work well on static scenes. While they exhibit some degree of robustness to moving objects, a large fraction of scenes is beyond the reach of current methods.
% These include cluttered dynamic scenes, including, for example, significant numbers of moving objects, regions of complex motion such as blowing foliage or large bodies of water, and complex reflectances from specular surfaces.
% While there is a need for new methods that are more robust to such non-static phenomena, the field also lacks good benchmarks for such challenging cases.  One of the main reasons is that it has been very difficult to get good ground truth camera trajectories and depth information.
Gathering ground truth camera trajectories is possible in restricted settings, e.g., with GPS, IMUs, and depth sensors, but such instrumentation is rarely available for in-the-wild data.

In this paper, we build such a benchmark by leveraging the growing availability of a special class of videos: 
panoramic videos captured by 360° cameras such as the \texttt{Ricoh~\!\!Theta} or the \texttt{Insta~\!\!360}. 
For each frame, such devices capture a full sphere of rays around the camera center.
Our insight is that, for such videos, structure-from-motion is much more reliable than for standard narrow field of view videos.
First, even if much of the scene contains difficult content~(e.g., dynamic elements), other parts of the scene are likely to be static, and therefore provide good features for camera motion estimation---in other words, stable features cannot ``hide'' from the camera, since the camera \textit{sees in all directions}.
Second, 360° videos have \textit{known intrinsics}, unlike regular in-the-wild videos where focal length may be unknown a priori.
Finally, pose estimation problems are fundamentally better posed when working with wide field-of-view imagery~\cite{levin2004visualodometry,levin2006motionuncertainty}.

In this paper, we describe how we built a custom SfM method for estimating camera poses for 360° videos, and used these as the basis for our new benchmark.
We first \textit{curate} a diverse set of 360° videos from the web, featuring moving cameras and difficult scene content~(moving objects, water, distractor objects fixed to the camera frame, and so on).
We run our new SfM method on these videos to estimate reliable camera poses as our pseudo-ground truth.
To form a difficult set of benchmark videos, we \textit{project} each 360° video to a perspective video that focuses on a difficult part of the scene, as measured by whether our baseline SfM method fails to accurately reproduce the pseudo-ground truth on that perspective crop.
We assemble a set of $100$ evaluation videos using this process, each up to roughly 30 seconds in length, 
% \ds{average \#frame and duration?} videos 
to form \textbf{\benchmark}, the \textbf{O}mnidirectional \textbf{R}econstruction \textbf{B}enchmark with \textbf{I}mage \textbf{T}rajectories.

% We use publicly available 360 videos on YouTube platform to curate a list of high quality and challenging set of scenes for benchmarking SFM methods. Our selection criteria aside from the quality of the videos is mainly movement of camera. The second criteria is the presence of moving objects and distractors in the scene while covering a variety of backgrounds and situations. In section ~\ref{sec:challenge} we explain the different challenges present in our benchmark evaluation set including camera rotation, scene lighting, texture, and visual distractors. 

% In order to curate an evaluation benchmark the ground truth trajectories has to be reliable with a form of guarantee that is independent of the performance of an already established SFM method. Recently, DynPose-100K was released with 100k clips, while being helpful for training purposes since there is no guarantee about the proposed estimated trajectories it is not suitable for evaluation purposes. 
% Also, the large size of the dataset makes it prohibitively expensive as an evaluation tool for diagnosing and developing new algorithms. Our proposed benchmark is relatively small so easier to use for evaluation while covering a diverse set of currently open challenges. Furthermore, we are able to provide guarantees about the accuracy of our ground truth trajectories due to independent estimations from non-overlapping viewpoints. 
% In a panoramic 360 capture while some portion of ones field of view may be dynamic and beyond the reach of current SfM methods, other portions of the 360 FOV may be sufficiently stable that current methods work well. We only accept a camera estimation as ground truth trajectory if two non-overlapping $90$ degree FOV trajectories match the $360$ bundle adjusted trajectory. 

We use \benchmark to evaluate a number of classic and recent SfM methods, including COLMAP~\cite{schonberger2016colmap}, MegaSaM~\cite{li2025megasam}, and VGGT~\cite{wang2025vggt}.
Our analysis shows that there is significant room for improvement among even such state-of-the-art techniques.
Every method fails to estimate camera poses well on \textit{at least} 20\% of clips. 
More importantly, the correlation between the failed clips of different methods is small.
Therefore, we find that \benchmark is a strong tool for diagnosing and evaluating current SfM methods, thus pointing the way towards future improvements.
% To foster innovation in this research area, upon acceptance, \ul{we commit to release our dataset and evaluation toolkit}.