MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection

Raman Dutt, Ondrej Bohdal, Pedro Sanchez, Sotirios Tsaftaris, Timothy Hospedales; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 4491-4501

Diffusion models excel in generating images that closely resemble their training data but are also susceptible to data memorization raising privacy ethical and legal concerns particularly in sensitive domains such as medical imaging. We hypothesize that this memorization stems from the overparameterization of deep models and propose that regularizing model capacity during fine-tuning can mitigate this issue. Firstly we empirically show that regulating the model capacity via Parameter-efficient fine-tuning (PEFT) mitigates memorization to some extent however it further requires the identification of the exact parameter subsets to be fine-tuned for high-quality generation. To identify these subsets we introduce a bi-level optimization framework MemControl that automates parameter selection using memorization and generation quality metrics as rewards during fine-tuning. The parameter subsets discovered through MemControl achieve a superior tradeoff between generation quality and memorization. For the task of medical image generation our approach outperforms existing state-of-the-art memorization mitigation strategies by fine-tuning as few as 0.019% of model parameters. Moreover we demonstrate that the discovered parameter subsets are transferable to non-medical domains. Our framework is scalable to large datasets agnostic to reward functions and can be integrated with existing approaches for further memorization mitigation. To the best of our knowledge this is the first study to empirically evaluate memorization in medical images and propose a targeted yet universal mitigation strategy. The code is available at https://github.com/Raman1121/Diffusion_Memorization_HPO