VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results

Hanwei Zhu, Haoning Wu, Zicheng Zhang, Lingyu Zhu, Yixuan Li, Peilin Chen, Shiqi Wang, Chris Wei Zhou, Linhan Cao, Wei Sun, Xiangyang Zhu, Weixia Zhang, Yucheng Zhu, Jing Liu, Dandan Zhu, Guangtao Zhai, Xiongkuo Min, Zhichao Zhang, Xinyue Li, Shubo Xu, Anh Dao, Yifan Li, Hongyuan Yu, Jiaojiao Yi, Yiding Tian, Yupeng Wu, Feiran Sun, Lijuan Jiao, Song Jiang; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025, pp. 3383-3393

Abstract

This paper presents a summary of the VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models (LMMs), hosted as part of the ICCV 2025 Workshop on Visual Quality Assessment. The challenge aims to evaluate and enhance the ability of state-of-the-art LMMs to perform open-ended and detailed reasoning about visual quality differences across multiple images. To this end, the competition introduces a novel benchmark comprising thousands of coarse-to-fine grained visual quality comparison tasks, spanning single images, pairs, and multi-image groups. Each task requires models to provide accurate quality judgments. The competition emphasizes holistic evaluation protocols, including 2AFC-based binary preference and multi-choice questions (MCQs). Around 100 participants submitted entries, with five models demonstrating the emerging capabilities of instruction-tuned LMMs on quality assessment. This challenge marks a significant step toward open-domain visual quality reasoning and comparison and serves as a catalyst for future research on interpretable and human-aligned quality evaluation systems.

Related Material

[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Zhu_2025_ICCV, author = {Zhu, Hanwei and Wu, Haoning and Zhang, Zicheng and Zhu, Lingyu and Li, Yixuan and Chen, Peilin and Wang, Shiqi and Zhou, Chris Wei and Cao, Linhan and Sun, Wei and Zhu, Xiangyang and Zhang, Weixia and Zhu, Yucheng and Liu, Jing and Zhu, Dandan and Zhai, Guangtao and Min, Xiongkuo and Zhang, Zhichao and Li, Xinyue and Xu, Shubo and Dao, Anh and Li, Yifan and Yu, Hongyuan and Yi, Jiaojiao and Tian, Yiding and Wu, Yupeng and Sun, Feiran and Jiao, Lijuan and Jiang, Song}, title = {VQualA 2025 Challenge on Visual Quality Comparison for Large Multimodal Models: Methods and Results}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops}, month = {October}, year = {2025}, pages = {3383-3393} }