\section{Building \benchmark}
\label{sec:method}

We extract our benchmark videos from online 360° panoramic videos.
On the web, such videos are plentiful, due to the popularity of cameras such as the Insta360 and Ricoh Theta.
First, we identify a challenging, yet diverse, set of 360° online video clips to form the basis of our camera pose estimation benchmark~(\Cref{sec:filtering}). 
Second, we estimate camera poses from these videos with an SfM pipeline customized for 360° video, and perform a cross-validation procedure to assess their viability as ground truth camera poses~(\Cref{sec:annotation}). 
Finally, we reproject these 360° videos to form challenging, interesting perspective videos, with a variety of fields of view and rotational behavior, while inheriting ground truth  estimates of camera position from the original 360° videos~(\Cref{sec:synthesizing}).
We depict the process we use to create ORBIT in~\Cref{fig:pipeline}.

\paragraph{Summary statistics}
After applying all of these pose estimation, filtering, and cropping stages, \benchmark consists of 100 clips of various lengths.
The clips are selected from 80 unique 360° videos, and have between 150 to 1,000 frames, with a frame rate of 30 fps. 
The clips have either an aspect ratio of $16{:}9$ or $9{:}16$, as these are the typical form factors found in consumer captures, with resolutions varying between $671 \times 377$ and $3356 \times 1888$, accompanied by ground truth camera trajectory.

\subsection{Identifying suitable videos}
\label{sec:filtering}
% 
% The first stage of filtering is based on the copyright selection of the video uploaders to include only YouTube videos that are permitted for our purposes.
We first find online 360° videos with a 1k minimum resolution, so that after reprojection into our desired perspective field-of-view videos we can attain at least VGA~(480p) compatible resolution.
We then look for videos that meet camera motion and camera continuation requirements.
In particular, we find that many online 360° videos are either a concatenation of static images, a concatenation of short clips, or a completely static camera observing a scene.
% \AT{see overleaf comment}
We manually select videos that feature a single camera moving across a long, continuous shot, and also identify videos whose content is likely challenging for camera pose estimation, including videos characterized by difficult environments (e.g., snow, water, sand, variable lighting, fast camera speeds, or scene dynamics).
We then break the videos into shorter $2000$ frame long clips without overlap to accommodate the current limitations of SFM and SLAM methods.

\subsection{Ground truth trajectory estimation}
\label{sec:annotation}
% 
Given each of our candidate videos, we obtain high-quality camera pose estimates (that serve as our ground truth when validated)
% \AT{wouldn't this be \ul{pseudo} ground-truth?} \DF{I think this can be called GT. GT is a quantity that serves as a target for an estimation procedure.  Typically it's obtained using measurement devices that are more accurate than one expects from the new method, often by using information not readily available at test time.}
by estimating robust camera poses with a specially designed 360° \textit{rig-based SfM} procedure~(\cref{sec:bundle}).
Given these camera poses, initially at an arbitrary scale, we convert them to approximately metric scale in order to make metrics, such as translation error, consistent for all videos~(\cref{sec:metricscale}).
We then validate our ground truth estimates by cross-validating SfM and SLAM camera estimates, dropping clips where cross validation fails to support the 360-based hypothetical ground truth~(\cref{sec:gtverify}).
% \AT{this sentence was mis-placed, I think (there is no subsection that matches...}
% \todo{Finally, in order to validate the poses, we reproject the 360° videos into \textit{four perspective viewpoints}, and estimate camera poses for each of them independently.}


% \AT{this feels misplaced?}
% \todo{Only 360° camera trajectories that agree with at least two other independent trajectories are selected as cross-validated ground truth.
% We now describe each of these steps in more detail.}
% \AT{need to explain why this ``at least two'' is a reasonable way to generate pseudo ground truth.}
% \SRBS{It may be worth clarifying why this was necessary.}
% \RS{I could say ``order to validate the poses'', but that was just said in the previous sentence.  Feel free to re-write this better.}

\input{assets/projection}

\subsubsection{Rig-based camera calibration}
\label{sec:bundle}
% 
Given a 360° video, we estimate per-frame 3D camera poses, along with a 3D point cloud, to jointly make use of all the information available in the input. 
The most common format for online 360° videos is an \emph{equirectangular} format, a spherical projection of the video where the position of a pixel in each frame is a function of the latitude and longitude of the ray corresponding to that pixel when projected onto a unit sphere centered on the camera position.
A number of methods for SfM/SLAM from 360° imagery have been proposed in the past.
One simple strategy for handling such videos is to adapt a standard SfM method like COLMAP to a new kind of camera model representing an equirectangular projection.
Such an approach involves a few challenges:
% 
\begin{enumerate*}[label=(\roman*)]
\item the equirectangular projection is \textit{highly anisotropic}, in that a small pixel offset at the equator of the projection corresponds to a much larger offset between 3D rays compared to the same offset near the poles; hence, reprojection error needs to be computed with care;
\item the \textit{projection wraps} (the left and right sides of the projection are connected), which implies that 3D points that project near the boundary must be handled with special care.
\end{enumerate*}

\paragraph{Cube-face sub-videos}
To avoid such difficulties, we take a different approach.  We treat each 360° video frame as having been captured by a rigidly-attached rig of multiple perspective cameras, where each camera corresponds to a perspective image projected from the raw equirectangular frame from a specific orientation.
Conceptually, one can think of reprojecting the equirectangular (spherical) projection to a cube map projection (six faces of a cube surrounding the center), and treating each cube face as an image captured by a perspective camera, rigidly attached to the cameras viewing the other faces.
This approach is inspired by prior work that treats panoramic reconstruction as handling images captured from separate cameras~\cite{uyttendaele2004imagebased, kangni2010orientation, klingner2013streetview}.
Given a video, we project each equirectangular video frame to four cube faces (front, back, left, right), as illustrated in~\Cref{fig:projection}.
We call these ``cube faces'', although, for added robustness, we use a 120° field of view, rather than 90°, meaning that each face \textit{overlaps} its neighbors.
We omit the top and bottom cube faces since they often either contain missing/spurious information~(e.g., a graphic is often pasted at the south pole of the video, while the north pole is often pointed at a featureless sky).
We therefore consider the original video to be broken into \textit{four}, temporally \textit{synchronized}, \textit{perspective} sub-videos.

\paragraph{Estimating the rig pose}
We run ORB-SLAM2~\cite{mur2015orb}, a standard SLAM pipeline, on the front-facing cube face as a way of bootstrapping the camera geometry for the video as a whole, since ORB-SLAM2 has detailed logic for initialization of 3D trajectories (for perspective videos) using keyframe selection.
Note that we use a modified version of ORB-SLAM2, following changes proposed by~\citet{zhou2018stereo} for improved accuracy, and restarting ORB-SLAM2 each time it loses track, therefore generating \textit{multiple} clips for each video.

\paragraph{Rig-based SfM}
For each such clip, we use its first $k{=}32$ ORB-SLAM2-derived
camera poses as \textit{initialization} for bundle adjustment.
This bundle adjustment stage recomputes correspondences across (and between) each sub-video using SIFT feature matching, then performs an incremental SfM procedure similar to COLMAP~\cite{schonberger2016colmap} to estimate poses for the rest of the video, periodically performing a full bundle adjustment.
This careful second-pass optimizes camera pose and 3D point geometry using all cube faces.
Further, it uses \textit{rig constraints} to ensure that each set of four cube face views is treated as a rigid collection of cameras with
\begin{enumerate*}[label=(\roman*)]
\item a shared center of projection, and
\item fixed relative orientations.
\end{enumerate*}
%
Note that we use ORB-SLAM2 and our rig-based 360° bundle adjustment pipeline as opposed to modern tools like VGGT~\cite{wang2025vggt} and MegaSaM~\cite{li2025megasam} since those recent methods are not straightforward to extend to 360° videos and would require retraining.

\paragraph{Output}
This procedure outputs a carefully optimized set of \textit{camera trajectories} for each 360° video, which we treat as the basis for our ground truth, along with a \textit{point cloud} annotated with frame visibility for each sequence.

\subsubsection{Metric scale calibration}
\label{sec:metricscale}
% 
Standard metrics for evaluating camera pose estimates, such as Absolute Trajectory Error (ATE), rely on the RMSE between estimated and ground-truth camera poses.
Hence, the \textit{scale} of ground truth poses is a key factor in such metrics.
Unfortunately, the SfM pipeline produces camera trajectories with arbitrary scale due to gauge ambiguity.

In order to attain interpretable, uniform metrics for camera pose accuracy, we therefore \textit{metrically re-scale} our pose estimates.
To perform this rescaling, we start from the four cube-face projected perspective sub-videos from the SfM stage of our pipeline. 
We use a state-of-the-art zero-shot monocular \textit{metric} depth estimation model, DepthPro~\cite{bochkovskii2024depth}, to predict metric depth for each frame of each sub-video.
We then reproject the visible 3D points calculated by our rig-based SfM stage into each sub-video and compute their depth ($z$-coordinate) with respect to the corresponding virtual camera. 

The ratio of this depth and the metric monocular depth at each pixel gives us an estimate of the scale factor that maps the SfM reconstruction to metric scale. 
For each frame, we take the median of the calculated scales as the frame scale. 
We then calculate the mean and standard deviation of the calculated metric frame scales across all frames. 
For robustness, we only consider clips where the standard deviation of frame scales is less than the mean.
For clips that pass this test, the average of all frame scales is accepted, and we apply that scale factor to map the entire clip to our approximately metric scale.

%\SSA{@Richard, we should expand here on details of parallax pipeline.}
%\NS{Now that there is more text above, this section is mostly repeating content from earlier. For instance we already talk about the ORB-SLAM2-based method. @Sara, maybe take a look at the prior part of this section and then see what makes sense to say here?}
%\NS{Actually, I'm confused---what is the difference between this paragraph and the "3.3. Cross-Validation section that comes next?}

% There are two main steps to consider when building the benchmark.
% \begin{itemize}
%     \item Calculating verifiable ground truth poses.
%     \item Selecting challenging or interesting test viewpoints.
% \end{itemize}

% Then for each period that verifiable ground truth exists we can select all the clips that result from projecting the 360 pano video into the perspective that satisfies the second condition.

\subsubsection{Ground-truth cross-validation}
\label{sec:gtverify}
% 
In order to further validate the trajectory estimates from the 360° rig, we \textit{cross-validate} by running ORB-SLAM2 independently on perspective crops around the cube faces, centered at $[90, 180, 270]$ degrees rotation from the frontal view~(we avoid the frontal cube face, as it was used to initialize the 360° estimates).
We employ Umeyama~\cite{umeyama1991least} to align the trajectories estimated from the 360° rig and from the perspective projection runs. 
Given the reference trajectory from the 360° rig, we use standard metrics from the SfM literature~\cite{zhang2018viotutorial}: Absolute Trajectory Error~(ATE) 
 and Relative Position Error~(RPE) to verify if any estimated perspective trajectory matches the 360° rig trajectory, where ATE and RPE are defined as follows (with separate RPE calculations for translation $t$ and rotation $R$):
 \begin{equation}
    % \ATE(g, e) = \RMSE_i(||g_i - e_i||_2^2)
    % \ATE(g, e) = \sqrt{\sum_i ||g_i - e_i||_2^2}
    \ATE(g, e) = \Big( \sum_i ||g_i - e_i||_2^2  \Big)^\frac{1}{2}\label{eq:ate}
\end{equation}
\begin{equation}
    % \RPE(g, e) = \RMSE_i \left( \Big|  \frac{||g_i \!-\! g_{i+1}||_2^2 - ||e_i \!-\! e_{i+1}||_2^2 } {||g_i - g_{i+1}||_2^2} \Big| \right)
   \!\! \RPEt(g, e) = \Big( \sum_i \Big|  \frac{||g_i \!-\! g_{i+1}||_2^2 - ||e_i \!-\! e_{i+1}||_2^2 } {||g_i - g_{i+1}||_2^2} \Big| \Big)^\frac{1}{2},\label{eq:rpe_t}
\end{equation}
\begin{equation}
\scriptsize
    % \RPE(g, e) = \RMSE_i \left( \Big|  \frac{||g_i \!-\! g_{i+1}||_2^2 - ||e_i \!-\! e_{i+1}||_2^2 } {||g_i - g_{i+1}||_2^2} \Big| \right)
   \!\! \RPEr(G, E) = 
   \Big( \sum_i \Big|\text{acos}\left(\frac{\text{Tr}((G_{ i}^{-1}G_{i+1})^{-1} (E_{i}^{-1}E_{i+1})) - 1}{2}\right) \Big| \Big)^\frac{1}{2},\label{eq:rpe_r}
\end{equation}
where $i$ denotes a frame index, $g$ and $e$ denote the positions, and $G$ and $E$ the rotation matrices along two camera trajectories in correspondence.

\begin{comment}
\at{Before thresholding, we also tweak ATE to account for the difference in speed of movement across sequences, as detailed in our \texttt{Supplementary Material}.}\AT{$\downarrow$}
% \RS{I agree with Andreas' new sentence that this is what we \emph{should} do, since we did it in the rig-face consistency check, but if we don't do it, we should remove this sentence and make a note to do this after the submission.}
\end{comment}
% \rit{I might have missed it but I don't think we use $g$ and $\mathbf{e}$ anywhere else in the paper----so no need to introduce these symbols?}
% fixed \rit comment by removing $g$ and \mathbf{e}

\begin{comment}
% \AT{agreed rick}
\paragraph{Metric adjustments}
Since the trajectory scale can vary greatly between different scenes with different camera speeds, ATE thresholds are scale dependent.
Therefore, we calculate a robust smoothed speed of movement between frames $\tau$ by taking the median of smoothed positional difference between position estimates.
If for a period of time between $(s, t)$, the rig trajectory $\mathbf{g}$ has a matching trajectory from an independent perspective crop viewpoint
$\mathbf{e}$, where $\ATE(\mathbf{g},\mathbf{e})<\tau(\mathbf{g})$ as well as ATE$(\mathbf{g},\mathbf{e})$, RPE-R$(\mathbf{g},\mathbf{e})$, and RPE-T$(\mathbf{g},\mathbf{e})$ are bellow strict thresholds, we consider $\mathbf{g}$ verified between $(s, t)$ ane we can use the ground truth pose estimate.
\RS{I just re-wrote this paragraph to the best of my ability, and I still find it unclear.  Should we just say that the details on how we decide if a rig-based pose estimate is verified w.r.t. the face-based estimates is given in the Supplementary material and then do a better job?}
\end{comment}

\subsection{Synthesizing benchmark videos}
\label{sec:synthesizing}
% Sarah's original description:
% Since our benchmark is targeted for perspective view cameras such as regular phone cameras, we design a method for cropping challenging perspective projections  for the frames that have verifiable ground truth trajectory. For each clip, we first select a cardinal rotational extrinsics around the $y$ axis. In the second step we introduce extra camera rotation, mainly along $y$ axis, to bring the rotational probability of our trajectories closer to a regular camera capture as opposed to a 360 capture. 
% Rick's new paragraph:
To generate benchmark videos that mimic the motion of regular cameras (e.g., smartphone or other handheld cameras), we vary the viewpoint direction of our perspective cropped videos rather than choosing a fixed orientation,  which often doesn't vary much in 360° videos.
If any of the perspective projection estimates fail the verification step in~\Cref{sec:gtverify}, we take that as an indicator of a challenging direction (i.e., ORB-SLAM2 has failed on that cube face). To maximize the potential challenge of our test trajectory, we select that cube face as the view for the initial frame. Otherwise, we start from the frontal cube face (i.e., $[0,0,0]$ rotation angles).
 
For subsequent frames, we add both a small per-frame rotational motion to \textit{simulate hand-held jitter}, and also a slowly varying rotation primarily around the vertical $y$-axis to simulate an observer looking around their environment.
\begin{comment}
\RS{To my eye, the amount of introduced jitter looks un-natural.  Also, most of the bounce I have observed in hand-held videos is medium frequency, i.e., 0.5-1 Hz, rather than per-frame 30Hz.
We can adjust this when polishing the final benchmark.}
\AT{+1, although shake shouldn't affect methods much (as most don't consider time anyway), it's mostly noticeable by us}
\end{comment}
% 
To select \textit{the field of view}, we choose a small random perturbation around 120°, since our videos are challenging enough without introducing further difficulties due to smaller fields of view.\footnote{In future work, we plan to vary the field of view across the clips and observe the performance of the algorithms being tested w.r.t. this parameter.}
\begin{comment}
\RS{I am post-hoc justifying the choice we made since I don't think we have time to change this and re-run everything, but we should for the final benchmark.  Also, we should copy this last sentence into Future work}
% Furthermore, we use a fixed intrinsic for each clip by randomly selecting an FOV and focal length for the clip.
\end{comment}

The final, perspective projected videos derived from the original 360° videos consistute our ORBIT benchmark. These perspective videos inherit ground truth camera poses from the 360° videos (with rotations suitably transformed according to the per-frame perturbations described above).