Enhanced Thermal-RGB Fusion for Robust Object Detection

Wassim El Ahmar, Yahya Massoud, Dhanvin Kolhatkar, Hamzah AlGhamdi, Mohammad Alja’afreh, Riad Hammoud, Robert Laganiere; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 365-374


Thermal imaging has seen rapid development in the last few years due to its robustness in different weather and lighting conditions and its reduced production cost. In this paper, we study the performance of different RGB-Thermal fusion methods in the task of object detection, and introduce a new RGB-Thermal fusion approach that enhances the performance by up to 9% using a sigmoid-activated gating mechanism for early fusion. We conduct our experiments on an enhanced version of the City Scene RGB-Thermal MOT Dataset where we register the RGB and corresponding thermal images in order to conduct fusion experiments. Finally, we benchmark the speed of our proposed fusion method and show that it adds negligible overhead to the model processing time. Our work would be useful for autonomous systems and any multi-model machine vision system. The improved version of the dataset, our trained models, and source code are available at https://github.com/wassimea/rgb-thermal-fusion

Related Material

@InProceedings{El_Ahmar_2023_CVPR, author = {El Ahmar, Wassim and Massoud, Yahya and Kolhatkar, Dhanvin and AlGhamdi, Hamzah and Alja{\textquoteright}afreh, Mohammad and Hammoud, Riad and Laganiere, Robert}, title = {Enhanced Thermal-RGB Fusion for Robust Object Detection}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, month = {June}, year = {2023}, pages = {365-374} }