FineControlNet: Fine-Level Text Control for Image Generation with Spatially Aligned Text Control Injection

Hongsuk Choi, Isaac Kasahara, Selim Engin, Moritz A. Graule, Nikhil Chavan-Dafle, Volkan Isler; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 3975-3984

Recently introduced ControlNet has the ability to steer the text-driven image generation process with geometric input such as human 2D pose or edge representations. While ControlNet provides control over the geometric form of the instances in the generated image it lacks the capability to dictate the visual appearance of each instance. We present FineControlNet to provide fine control over each instance's appearance while maintaining the pose control capability. Specifically we develop and demonstrate FineControlNet with geometric control via human pose images and appearance control via instance-level text prompts. The spatial alignment of instance-specific text prompts and 2D poses in latent space enables the fine control capabilities of FineControlNet. We evaluate the performance of FineControlNet with rigorous comparison against state-of-the-art pose-conditioned text-to-image diffusion models. FineControlNet achieves superior performance in generating high quality images that follow instance-specific controls. We will release the code and the dataset.