Multi-Modal Large Language Model driven Augmented Reality Situated Visualization: the Case of Wine Recognition

Vincenzo Armandi, Andrea Loretti, Lorenzo Stacchio, Pasquale Cascarano, Gustavo Marfia; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 1313-1322

Situated Visualizations (SV) and reality-based information retrieval systems enhanced by Mixed Reality (MR) and Augmented Reality (AR) enable the overlay of digital information onto real-world objects providing context-aware content through computer vision. Despite their potential these systems face significant challenges in scalability and adaptability particularly for domains like wine recognition where diverse label designs frequent updates and limited historical databases complicate automated analysis. SOLLAMA (SOmmeLier LlAMA) is a novel wine recognition framework designed to address the scalability and adaptability challenges of augmented reality systems in recognizing diverse wine labels. Leveraging Multimodal Large Language Models (MLLMs) SOLLAMA integrates visual and textual cues for accurate label interpretation bypassing the need for extensive image datasets and traditional OCR methods. Built on the Augmented Wine Recognition (AWR) system it replaces the OCR module with LLAMA 3.2 for advanced text recognition and contextual understanding. Key benefits include scalability across diverse designs and simplified server-free deployment. Experimental validation on a dataset of wine labels from Italy's Emilia-Romagna region highlights the system's effectiveness demonstrating its potential to transform wine recognition in AR-based applications.