Constellations: A Novel Dataset for Studying Iterative Inference in Humans and AI

Tarun Khajuria, Kadi Tulver, Taavi Luik, Jaan Aru; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022, pp. 5142-5152

Under complex viewing conditions, human perception relies on generating hypotheses and revising them in an iterative fashion. We developed novel visual stimuli to study such iterative inference in humans and AI. In these stimuli, called "constellations", all local information about the object has been removed and it can only be recognized when taking into account the global pattern. We here describe the dataset and demonstrate that humans indeed use an iterative process of generating hypotheses and refining them to solve these images. We also provide code that allows researchers to create their own constellation images. The constellation dataset allows researchers to develop sketching algorithms for guessing the hidden object. As such algorithms used by humans appear to be iterative in nature, this dataset will facilitate the study of iterative inference in minds and machines.