Dense Trajectory Fields: Consistent and Efficient Spatio-Temporal Pixel Tracking

Marc Tournadre, Catherine Soladié, Nicolas Stoiber, Pierre-Yves Richard; Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 2212-2230

In this paper, we present Dense Trajectory Fields (DTF), a novel low-level holistic approach inspired by optical-flow and trajectory approaches, focusing on both spatial and temporal aspects at once. DTF contains the dense and long-term trajectories of all pixels from a reference frame, over an entire input sequence. We solve it through DTF-Net, a fast and lightweight neural network, comprising 3 main components: (1) a joint iterative refinement of image and motion features over residual layers, (2) token-based Reciprocal Attention clusters and, (3) a Refinement Network that builds patch-to-patch cost-volumes around salient centroid trajectories. We extend the recent Kubric dataset to provide dense ground-truth over all pixels, to train our network. We conduct experiments showing that usual optical-flow and trajectory methods exhibit inconsistencies either temporally or spatially, where DTF-Net offers a better compromise while keeping faster, giving a coherent motion over the entire sequence.