LLM-RSPF: Large Language Model-Based Robotic System Planning Framework for Domain Specific Use-Cases

Chandan Kumar Singh, Devesh Kumar, Vipul Sanap, Rajesh Sinha; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 7277-7286

The employment of large language models (LLMs) for task planning and reasoning has emerged as a focal point of interest within the robotics research community. However directly applying LLMs even with large token-sized prompts does not achieve the task planning performance required for an industrial-grade domain-specific use-case (DSU). This work aims to overcome the obstacles of a robotic task planner for DSUs by introducing a novel planning framework LLM-RSPF (Large Language Model-based Robotic System Planning Framework). Central to the LLM-RSPF is a novel robotic system ontology that organizes the components of the robotic system in a coherent and a systematic manner. The ontology empowers the LLM-RSPF to efficiently capture a contextual representation of the DSU using the LLMs. Subsequently the research introduces a LLM-tuning regimen referred as chain of hierarchical thought (CoHT) specifically crafted to complement the proposed system ontology. Integrating these two components the LLM-RSPF aims to enhance the accuracy robustness and throughput of a robotic system in a cost-effective manner. In addition the research presents an empirical methodology to generate the LLM-tuning dataset size for a guaranteed performance. The LLM-RSPF is validated on a retail order-fulfillment use-case thereby illustrating the efficacy of the framework. Through rigorous evaluation the LLM-RSPF demonstrates exceptional performance on the generated dataset effectively meeting the DSU objectives.