A Data-Driven Approach Based on Dynamic Mode Decomposition for Efficient Encoding of Dynamic Light Fields

Joshitha Ravishankar, Sally Khaidem, Mansi Sharma; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 3447-3453

Dynamic light fields provide a richer, more realistic 3D representation of a moving scene. However, this leads to higher data rates since excess storage and transmission requirements are needed. We propose a novel approach to efficiently represent and encode dynamic light field data for display applications based on dynamic mode decomposition (DMD). Acquired images are firstly obtained through optimized coded aperture patterns for each temporal frame/camera viewpoint of a dynamic light field. The underlying spatial, angular, and temporal correlations are effectively exploited by a data-driven DMD on these acquired images arranged as time snapshots. Next, High Efficiency Video Coding (HEVC) removes redundancies in light field data, including intra-frame and inter-frame redundancies, while maintaining high reconstruction quality. The proposed scheme is the first of its kind to treat light field videos as mathematical dynamical systems, leverage on dynamic modes of acquired images, and gain flexible coding at various bitrates. Experimental results demonstrate our scheme's superior compression efficiency and bitrate savings compared to the direct encoding of acquired images using HEVC codec.