Adversarial Weather-Resilient Image Retrieval: Enhancing Restoration using Captioning for Robust Visual Search

Prem Shanker Yadav, Kushall Singh, Dr.Dinesh Kumar Tyagi, Dr.Ramesh Babu Battula; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2024, pp. 129-142

Accurate image retrieval in real-world scenarios is often hampered by degraded or noisy images, particularly those affected by adverse weather conditions such as rain, fog, or snow. Traditional retrieval methods that rely solely on feature extraction struggle to handle these degraded inputs and image captioning models are similarly limited in their ability to interpret distorted images. To address these challenges, we propose a novel framework that integrates image restoration with image captioning to create a robust image retrieval system capable of handling images degraded by adverse weather. Additionally, we introduce an integrated loss function to optimize restoration and captioning processes for degraded images. Our system enhances retrieval performance in challenging weather conditions by leveraging improved visual content alongside semantic context. Evaluations on Flicker8k dataset demonstrate that our approach significantly outperforms traditional image retrieval systems, particularly in scenarios where weather-induced degradation presents a challenge.