Gatha: Relational Loss for Enhancing Text-Based Style Transfer

Surgan Jandial, Shripad Deshmukh, Abhinav Java, Simra Shahid, Balaji Krishnamurthy; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 3546-3551

Text-based style transfer is a promising area of research that enables the generation of stylistic images from plain text descriptions. However, the existing text-based style transfer techniques do not account for the subjective nature of prompt descriptions or the nuances of style-specific vocabulary during the optimization process. This severely limits the stylistic expression of the predominant models. In this paper, we address this gap by proposing Gatha, which incorporates subjectivity by introducing an additional loss function that enforces the relationship between stylized images and a proxy style set to be similar to the relationship between the text description and the proxy style set. We substantiate the effectiveness of Gatha through both qualitative and quantitative analysis against the existing state-of-the-art models and show that our approach allows for consistently improved stylized images.