Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents

Shivansh Patel, Saim Wani, Unnat Jain, Alexander G. Schwing, Svetlana Lazebnik, Manolis Savva, Angel X. Chang; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15953-15963

Abstract


Communication between embodied AI agents has received increasing attention in recent years. Despite its use, it is still unclear whether the learned communication is interpretable and grounded in perception. To study the grounding of emergent forms of communication, we first introduce the collaborative multi-object navigation task 'CoMON.' In this task, an 'oracle agent' has detailed environment information in the form of a map. It communicates with a 'navigator agent' that perceives the environment visually and is tasked to find a sequence of goals. To succeed at the task, effective communication is essential. CoMON hence serves as a basis to study different communication mechanisms between heterogeneous agents, that is, agents with different capabilities and roles. We study two common communication mechanisms and analyze their communication patterns through an egocentric and spatial lens. We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Patel_2021_ICCV, author = {Patel, Shivansh and Wani, Saim and Jain, Unnat and Schwing, Alexander G. and Lazebnik, Svetlana and Savva, Manolis and Chang, Angel X.}, title = {Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {15953-15963} }