DermAI: A Chatbot Assistant for Skin lesion Diagnosis Using Vision and Large Language Models

Viet-Tham Huynh, Trong-Thuan Nguyen, Thao Thi-Phuong Dao, Tam V. Nguyen, Minh-Triet Tran; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2024, pp. 287-301

In dermatology, the demand for accurate skin lesion diagnoses is critical, especially during peak times like summer when skin cancer screenings surge. The need for efficient processing of large volumes of medical images and the risk of human error highlights the importance of innovative diagnostic tools. This paper introduces DermAI, an advanced AI-driven framework to improve diagnostic accuracy and efficiency in skin lesion analysis. DermAI combines a state-of-the-art segmentation model and a large language model to assist clinicians in interpreting medical images swiftly and precisely. Our framework isolates and analyzes key lesion features using advanced segmentation models and vision encoders, while a GPT-4-based language model provides contextual insights to better understand lesion characteristics and potential malignancies. By integrating visual and linguistic analysis, DermAI reduces diagnostic errors, alleviates clinician workloads, and enhances patient care with faster, more accurate results, supporting dermatologists in making informed decisions and advancing AI-assisted diagnostics.