IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models

Siying Cui, Jia Guo, Xiang An, Jiankang Deng, Yongle Zhao, Xinyu Wei, Ziyong Feng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 950-959

Leveraging Stable Diffusion for the generation of personalized portraits has emerged as a powerful and noteworthy tool enabling users to create high-fidelity custom character avatars based on their specific prompts. However existing personalization methods face challenges including test-time fine-tuning the requirement of multiple input images low preservation of identity and limited diversity in generated outcomes. To overcome these challenges we introduce IDAdapter a tuning-free approach that enhances the diversity and identity preservation in personalized image generation from a single face image. IDAdapter integrates a personalized concept into the generation process through a combination of textual and visual injections and a face identity loss. During the training phase we incorporate mixed features from multiple reference images of a specific identity to enrich identity-related content details guiding the model to generate images with more diverse styles expressions and angles. Extensive evaluations demonstrate the effectiveness of our method achieving both diversity and identity fidelity.