PatchFlow: A Two-Stage Patch-Based Approach for Lightweight Optical Flow Estimation

Ahmed Hammad Alhawwary, Janne Mustaniemi, Janne Heikkila; Proceedings of the Asian Conference on Computer Vision (ACCV), 2022, pp. 3740-3756

The deep learning-based optical flow methods have shown noticeable advancements in flow estimation. The dense optical flow map offers high flexibility and quality for aligning neighbouring video frames. However, they are computationally expensive, and the memory requirements for processing high-resolution images such as 2K, 4K and 8K on resources-limited devices such as mobile phones can be prohibitive. We propose a patch-based approach for optical flow estimation. We redistribute the regular CNN-based optical flow regression into a two-stage pipeline, where the first stage estimates an optical flow for a low-resolution image version. The pre-flow is input to the second stage, where the high-resolution image is partitioned into small patches for optical flow refinement. With such a strategy, it becomes possible to process high-resolution images when the memory requirements are not sufficient. On the other hand, this solution also offers the ability to parallelize the optical flow estimation when possible. Furthermore, we show that such a pipeline can additionally allow for utilizing a lighter and shallower model in the two stages. It can perform on par with FastFlowNet (FFN) while being 1.7x faster computationally and with almost a half of the parameters. Against the state-of-the-art optical flow methods, the proposed solution can show a reasonable accuracy trade-off for running time and memory requirements.