CoDISP: Exploring Compressed Domain Camera ISP with RGB-guided Encoder

Molin Zhang, Soumendu Majee, Chengyu Wang, Seok-Jun Lee, Hamid Sheikh; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5878-5888

Most mobile device Image Signal Processing (ISP) pipelines operate directly on RAW image data for all processing tasks. However the rise of super-high-resolution cameras on mobile devices has led to increased memory demands for multi-frame ISP pipelines. In this work we introduce a novel ISP pipeline that operates on a learned compressed domain aiming to conserve memory for downstream ISP modules' inputs. We utilize RGB image compression to define a compressed latent domain preserving both semantic information and high-frequency details. To facilitate mapping of raw images to the compressed domain we develop a transfer learning strategy. All downstream processing tasks including demosaicing single and multi-frame denoising and registration are performed on this compressed latent domain. We demonstrate the effectiveness of our compressed domain ISP pipeline on both public and internal datasets. Remarkably our pipeline achieves ISP performance similar to non-compression methods while significantly reducing mobile memory requirements.