Low-Rank Residual Diffusion Models

Junfu Tan, Jiang Yuan; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 35747-35757

Residual diffusion models have achieved remarkable progress in image restoration tasks. In near-domain transformations such as image deraining, however, we observe that the residuals exhibit an inherent low-rank structure. Motivated by this property, we propose the Low-Rank Residual Diffusion Model (LRDM), which performs diffusion within a compact low-rank residual subspace for efficient and structure-preserving restoration. We formalize this observation as the Low-Rank Residual Assumption and show that the variational lower bound becomes tighter when residuals lie in a low-rank space. Building on this insight, we introduce an Asymmetric Residual Diffusion Process that constrains the forward process in the low-rank domain while maintaining full-rank flexibility in the reverse process. To accommodate the varying complexity of residuals across diffusion timesteps, we further introduce an Adaptive Rank Selection mechanism that dynamically adjusts the rank during the diffusion process. Experiments on deraining, deblurring, and deshading benchmarks show that LRDM surpasses full-rank diffusion baselines and achieves state-of-the-art performance, validating the advantage of modeling diffusion in a low-rank residual space. Project Page: \urlstyle same \hypersetup allcolors=magenta https://github.com/JF-Tan/LRDM .