Abstract Visual Reasoning Enabled by Language

Giacomo Camposampiero, Loïc Houmard, Benjamin Estermann, Joël Mathys, Roger Wattenhofer; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 2642-2646


While artificial intelligence (AI) models have achieved human or even superhuman performance in many well-defined applications, they still struggle to show signs of broad and flexible intelligence. The Abstraction and Reasoning Corpus (ARC), a visual intelligence benchmark introduced by Francois Chollet, aims to assess how close AI systems are to human-like cognitive abilities. Most current approaches rely on carefully handcrafted domain-specific program searches to brute-force solutions for the tasks present in ARC. In this work, we propose a general learning-based framework for solving ARC. It is centered on transforming tasks from the vision to the language domain. This composition of language and vision allows for pre-trained models to be leveraged at each stage, enabling a shift from handcrafted priors towards the learned priors of the models. While not yet beating state-of-the-art models on ARC, we demonstrate the potential of our approach, for instance, by solving some ARC tasks that have not been solved previously.

Related Material

@InProceedings{Camposampiero_2023_CVPR, author = {Camposampiero, Giacomo and Houmard, Lo{\"\i}c and Estermann, Benjamin and Mathys, Jo\"el and Wattenhofer, Roger}, title = {Abstract Visual Reasoning Enabled by Language}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, month = {June}, year = {2023}, pages = {2642-2646} }