Advancing Chart Question Answering with Robust Chart Component Recognition

Hanwen Zheng, Sijia Wang, Chris Thomas, Lifu Huang; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 5741-5750

Chart comprehension presents significant challenges for machine learning models due to the diverse and intricate shapes of charts. Existing multimodal methods often overlook these visual features or fail to integrate them effectively for Chart Question Answering. To address this we introduce ChartFormer a unified framework that enhances chart component recognition by accurately identifying and classifying components such as bars lines pies titles legends and axes. Additionally we propose a novel Question-guided Deformable Co-Attention (QDCAt) mechanism which fuses chart features encoded by ChartFormer with the given question leveraging the question's guidance to ground the correct answer. Extensive experiments demonstrate a 3.2% improvement in mAP over the baselines for chart component recognition. For ChartQA and OpenCQA tasks our approach achieves improvements of 15.4% in accuracy and 0.8 in BLEU score respectively underscoring the robustness of our solution for detailed visual data interpretation across various applications. The source code and dataset are publicly available at https://github.com/VT-NLP/chartQA.