SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations

Jamie Menjay Lin, Jisoo Jeong, Hong Cai, Risheek Garrepalli, Kai Wang, Fatih Porikli; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2162-2171

Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements achieving real-time on-device optical flow estimation remains a complex challenge. First an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ensure real-time performance on devices. Second the necessity for real-time on-device operation imposes constraints that weaken the model's capacity to adequately handle ambiguities in flow estimation thereby intensifying the difficulty of preserving flow accuracy. This paper introduces two synergistic techniques Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL) designed to enhance the capabilities of optical flow models with a focus on addressing optical flow regression ambiguities. These techniques prove particularly effective in mitigating error propagation a prevalent issue in optical flow models that employ iterative refinement. Notably these techniques add negligible to zero overhead in model parameters and inference latency thereby preserving real-time on-device efficiency. The effectiveness of our proposed SCI and RFL techniques collectively referred to as SciFlow for brevity is demonstrated across two distinct lightweight optical flow model architectures in our experiments. Remarkably SciFlow enables substantial reduction in error metrics (EPE and Fl-all) over the baseline models by up to 6.3% and 10.5% for in-domain scenarios and by up to 6.2% and 13.5% for cross-domain scenarios on the Sintel and KITTI 2015 datasets respectively.