Looking at Words and Points with Attention: a Benchmark for text-to-Shape Coherence

Andrea Amaduzzi, Giuseppe Lisanti, Samuele Salti, Luigi Di Stefano; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 2868-2877

Abstract


While text-conditional 3D object generation and manipulation have seen rapid progress, the evaluation of coherence between generated 3D shapes and input textual descriptions lacks a clear benchmark. The reason is twofold: a) the low quality of the textual descriptions in the only publicly available dataset of text shape pairs; b) the limited effectiveness of the metrics used to quantitatively assess such coherence. In this paper, we propose a comprehensive solution that addresses both weaknesses. Firstly, we employ large language models to automatically refine textual descriptions associated with shapes. Secondly, we propose a quantitative metric to assess text-to-shape coherence, through cross-attention mechanisms. To validate our approach, we conduct a user study and compare quantitatively our metric with existing ones. The refined dataset, the new metric and a set of text-shape pairs validated by the user study comprise a novel, fine-grained benchmark that we publicly release to foster research on text-to-shape coherence of text-conditioned 3D generative models. Benchmark available at https://cvlab-unibo.github.io/CrossCoherence-Web/.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Amaduzzi_2023_ICCV, author = {Amaduzzi, Andrea and Lisanti, Giuseppe and Salti, Samuele and Di Stefano, Luigi}, title = {Looking at Words and Points with Attention: a Benchmark for text-to-Shape Coherence}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops}, month = {October}, year = {2023}, pages = {2868-2877} }