Compressed Domain Multiframe Processing

Chengyu Wang, Jing Li, Saurabh Kumar, Seok-Jun Lee, Hamid Sheikh; Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, 2025, pp. 1954-1963

Multiframe processing has become an essential component in mobile devices to produce images with better qualities, such as reduced noise and improved dynamic range. However, processing multiple frames poses challenges in system memory and computation power, especially for high resolution images. In this work we present a compressed domain multiframe processing pipeline that operates in a compressed domain defined by an encoder-decoder and vector quantization. The encoder-decoder learns the features from the raw frame and produces the RGB image. Vector quantization is used to quantize the feature to achieve compression. In this compressed domain we show common multiframe processing functions, including demosaicing, denoising, image registration, image deghosting and HDR blending. Experiments on real mobile captures demonstrate the effectiveness of the proposed compressed domain multiframe processing pipeline. The proposed method achieves image quality similar to non-compression methods with less memory and computation requirement.