Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models

Liu, Nan; Du, Yilun; Li, Shuang; Tenenbaum, Joshua B.; Torralba, Antonio

Nan Liu, Yilun Du, Shuang Li, Joshua B. Tenenbaum, Antonio Torralba; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 2085-2095

Abstract

Text-to-image generative models have enabled high-resolution image synthesis across different domains, but require users to specify the content they wish to generate. In this paper, we consider the inverse problem - given a collection of different images, can we discover the generative concepts that represent each image? We present an unsupervised approach to discover generative concepts from a collection of images, disentangling different art styles in paintings, objects, and lighting from kitchen scenes, and discovering image classes given ImageNet images. We show how such generative concepts can accurately represent the content of images, be recombined and composed to generate new artistic and hybrid images, and be further used as a representation for downstream classification tasks.

Related Material

[pdf] [supp] [arXiv]

[bibtex]

@InProceedings{Liu_2023_ICCV, author = {Liu, Nan and Du, Yilun and Li, Shuang and Tenenbaum, Joshua B. and Torralba, Antonio}, title = {Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2023}, pages = {2085-2095} }