SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving

Xuewen Luo, Chenxi Liu, Fan Ding, Fengze Yang, Yang Zhou, Junnyong Loo, Hwa Hui Tew; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 989-996

This study addresses the critical need for enhanced situational awareness in autonomous driving (AD) by leveraging the contextual reasoning capabilities of large language models (LLMs). Unlike traditional perception systems that rely on rigid label-based annotations it integrates real-time multimodal sensor data into a unified LLMs-readable knowledge base enabling LLMs to dynamically understand and respond to complex driving environments. To overcome the inherent latency and modality limitations of LLMs a proactive retrieval augmented generation (RAG) is designed for AD combined with a chain-of-thought prompting mechanism ensuring rapid and context-rich understanding. Experimental results using real world V2X datasets demonstrate significant improvements in perception and prediction performance highlighting the potential of this framework to enhance safety adaptability and decision-making in next-generation AD systems.