
\section{Results}
\input{assets/samples}

In this section, we perform a quantitative evaluation of several leading algorithms and discuss some of the main challenging issues.

\subsection{Challenges} 
\label{sec:challenge}
Most SfM methods rely on stable keypoints or textures that can be tracked over different frames. Therefore, two key types of challenges include (1) a lack of stable keypoints and (2) the presence of outlier or moving keypoints. Most of the clips in our benchmark include one or both of these challenges. Nearly all of our clips, more than $90\%$, have moving objects present in all or part of the frames. We further categorize the challenges present in the videos as follows:

\paragraph{Low texture in static regions} Our benchmark includes videos that are captured in snow, in desert, and under water. Approximately $10\%$  of our clips are situated in areas without definitive texture or distinctive keypoints. Our GT validation technique can occasionally successfully find clips with this criteria given that cube faces can have different textures (mountains alongside sand or forests alongside water), which often will not be visible in the test viewpoint.

\paragraph{Dark scenes} Another scenario that affects the clarity of the scenes and hinders SfM algorithms are dark, low-contrast scenes. Our benchmark contains scenes that are captured either during nighttime or in dark areas like caves. These scenes also constitute approximately $10\%$ of our clips.

\paragraph{High camera speed} Large displacements from one frame to the next break certain assumptions made in SLAM-based methods. Based on the camera mounting method, high camera speed may also lead to jittery videos, which we have found to be especially challenging to methods that rely on smoothness of frame-to-frame motion. Some of our scenes are captured from inside moving vehicles, while paragliding, while skiing, or in otherwise fast-moving scenarios. Errors in these scenes also result in higher ATE error values due to larger ground truth trajectory movements.

\paragraph{Camera rotation} Since our clips are selected from 360° videos we have the freedom of selecting camera rotation at each frame. Therefore, for every second (30th frame) we sample the rotation angle of the camera $(x,y,z)$ from $N([0,0,0], [1,20,2])$ and we use spherical linear interpolation to assign smoothly varying camera angles to in-between frames. We noticed that some methods such as MegaSaM are more prone to error in the presence of large camera rotations.

\paragraph{Camera parallel objects} An especially challenging case involving moving objects occurs when the moving object is in a constant position with respect to camera. 
Many such examples can be found in real-world situations, e.g., while capturing ego-motion such as a boat or a car that the camera itself is riding on. 
In these cases there can be two frames of reference for the position of camera, one with respect to the world and one with respect to the object that is moving alongside of camera (and note that we always choose the world frame of reference, meaning the coordinate frame of the static Earth). 
We observe more failures for simpler methods such as COLMAP in these cases.

\paragraph{Crowded scenes} About $10\%$ of our clips are captured in crowded areas such as busy streets and tourist sites. In these cases 
% there are many moving objects and 
a significant area of each frame is occluded by moving objects, reducing the visibility of the static and reliably trackable portions of the scene.

\paragraph{Fluid texture} Large bodies of water are an extreme case of dynamic scenes since the water is in constant motion while having a repeatable texture. 
The fluidity of water introduces many noisy matches between one frame and the next which hinders methods that lack a strong robustness criterion.

\subsection{Failure modes}
One critical aspect when comparing the performance of different SfM methods is to account for the possibility of failing to output any camera estimates. 
Some SfM methods, such as MegaSaM and feedforward approaches like VGGT, always output a camera estimate (no matter how inaccurate). 
However, other methods, such as COLMAP and ParticleSfM, can sometimes drop a few frames, or can at times completely fail on a clip and produce no output whatsoever. 
% 
The reasons behind such failures include the inability to finding sufficient reliable matches, in some or all frames. 
In our evaluations, if some of the frames are dropped, we assume the previously reported camera pose for the dropped frames until a new estimate is reported.

% \RS{New text based on Wednesday night's discussion between Rick and Sara:}
To quantify the failure rate of various algorithms, beyond ATE (Eq.~\ref{eq:ate}) and RPE (~\cref{eq:rpe_t,eq:rpe_r}), we set a threshold on the ATE for each clip and consider it to have failed when the ATE exceeds this threshold (in addition to total failure to produce an output in the case of methods like COLMAP).
The average failure rates for each algorithm are reported in the last column of Table~\ref{tab:benchmark}.
\begin{comment}
\RS{Sara has generated plots failure rates vs. ATE and RPE thresholds,
so this figure should be inserted and referenced.
Note that when deciding whether at 360° rig pose matches a cube-face pose, Sara uses a local velocity estimate $\tau$ to compensate for various camera speeds, but here, only a single threshold is used across all videos.}
\end{comment}

\input{assets/benchmark}

\subsection{Evaluations}
We evaluate several methods, including state-of-the-art SfM methods, using our benchmark in Table~\ref{tab:benchmark}. The current gold standard in many SfM applications, including novel view synthesis, is COLMAP \cite{schonberger2016colmap} due to its reliability and use of careful incremental bundle adjustment to produce highly accurate camera poses. 
Therefore, as a strong baseline we consider COLMAP performance on our benchmark. 

\begin{itemize}[listparindent=0pt,leftmargin=*,itemindent=0pt]
\item ORB-SLAM \cite{mur2015orb} is a leading SLAM method based on traditional, handcrafted (fast) feature matching approaches.
% that uses tracking techniques to improves the robustness of the camera estimations in the presence of moving objects. 
% Conversely if a few frames are not trackable it cannot stitch together the rest of the frames with the previous frames and will drop the other frames.
It can drop frames if it cannot reliably track them.

\item MonST3R \cite{zhang2024monst3r} extends DUSt3R \cite{Wang2024dust3r} to include dynamic scenes by projecting onto 3D point clouds estimated from pairs of frames.

\item MegaSaM \cite{li2025megasam} is a recent state-of-the-art method that has gained traction due to its excellent performance on available benchmarks. MegaSaM is based on the popular DROID-SLAM learned bundle adjustment framework~\cite{Teed2021droidslam}, and utilizes monocular depth estimation, along with predicted motion probabilities, within a robust differentiable bundle adjustment.

\item RoMo \cite{goli2025romo} is also a recent method for zero-shot motion segmentation for SFM applications. RoMo produces motion masks that can be used as an input to other SFM methods. Here we combined the masks with MegaSaM as the best available method on our benchmark.

\item VGGT \cite{wang2025vggt} is a leading approach among the recent crop of feedforward methods that train neural network to directly predict camera poses and 3D point maps from a set of input frames. VGGT-Long \cite{deng2025vggt} is a recent method that extends VGGT to clips with more than $200$ frames, the limit of VGGT. Since our benchmark has often more than $200$ frames, we report the results of VGGT-Long.
\end{itemize}

We show the performance of all of these methods on ORBIT in Table.~\ref{tab:benchmark}. 
This table presents Absolute Trajectory Error (ATE), as well as relative errors for both camera translation (RPE-T) and rotation (RPE-R).
The results show that all methods exhibit varying degrees of poor performance on \benchmark. Some techniques including COLMAP and ORB-SLAM2 fail completely on some clips and overall all measurements show a significantly large standard deviation, which point to the fact that average ATE, RPE-T, and RPE-R are not informative for the SFM task and specially on our benchmark.

We also define a binary success metric and report success rates as a more uniform method of evaluation. 
For our main success metric we define a threshold of $\text{ATE} < 0.5,~\text{RPE-R} < 0.4,~\text{RPE-T} < 2.0$ which is 2$\times$ larger than the thresholds we used for cross validation in Sec.~\ref{sec:gtverify}. 
We also report a more relaxed success criteria by again doubling these thresholds. 
Due to the highly varied performance distribution in a clip-by-clip basis, the best performing method according to each metric can differ. 
While COLMAP has the worst average ATE, it has a $25.27\%$ success rate in the most strict matching criteria. This suggest that COLMAP is still a good choice in less challenging application scenarios.
\input{assets/ate_success}

\input{assets/correlation}

We further investigate the distribution of clip ATEs for each method by both considering the staggered success ratio of varying ATE thresholds in Figure~\ref{fig:ate_success}. We find that while feedforward methods such as MonST3R rarely match the ground truth trajectory as perfectly as a bundle adjustment--based method such as COLMAP and MegaSaM, they have a lower upper bound on their error. 
Another interesting finding is that the main effect of combining RoMo masks with MegaSaM is to improve the performance on the high error clips, while preserving performance on less challenging clips. 

One of the main advantages of \benchmark is its concise diversity of challenges. To measure the diversity of our clips we calculate the correlation matrix between the success and failure rates of different methods. Figure~\ref{fig:correlation} shows the low correlation between these methods' performance on \benchmark. Apart from MegaSaM and RoMo+MegaSaM which unsurprisingly have a high correlation of $0.80$, the other pairings have little correlation. This indicates that ORBIT is a successful tool for diagnosing failure modes of models, rather than a mere selection of unreasonably difficult clips. As of note, combining RoMo with MegaSaM increases the correlation of results with COLMAP and MonST3R. %Hence, the masking technique increases the similarity of performance between MegaSAM and Colmap and Monst3r.
