Can Out-of-Domain Data Help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?

Amartya Bhattacharya, Debarshi Brahma, Suraj Nagaje, Anmol Asati, Vikas Verma, Soma Biswas; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 2808-2817

Spread of fake news using out-of-context images and captions has become widespread in this era of information overload. Since fake news can belong to different domains like politics sports etc. with their unique characteristics inference on a test image-caption pair is contingent on how well the model has been trained on similar data. Since training individual models for each domain is not practical we propose a novel framework termed DPOD (Domain-specific Prompt tuning using Out-of-Domain data) which can exploit out-of-domain data during training to improve fake news detection of all desired domains simultaneously. First to compute generalizable features we modify the Vision-Language Model CLIP to extract features that help to align the representations of the images and corresponding captions of both the in-domain and out-of-domain data in a label-aware manner. Further we propose a domain-specific prompt learning technique which leverages training samples of all the available domains based on the extent they can be useful to the desired domain. Extensive experiments on the large-scale NewsCLIPpings and VERITE benchmarks demonstrate that DPOD achieves state of-the-art performance for this challenging task. Code: https://github.com/scviab/DPOD.