WARMOS: Enhancing Weather-Affected Referred Moving Object Segmentation

Prafulla saxena, Dinesh Kumar Tyagi, Santosh Kumar Vipparthi, Subrahmanyam Murala; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2024, pp. 102-114

Environmental noise, such as haze and rain, poses significant challenges in video surveillance, in tasks like referred video object segmentation. These weather-related disturbances introduce excessive pixel variance, making moving object segmentation more complex. In this work, we focus on addressing the issue of adverse weather conditions by simulating the effects of haze and rain in videos and employing a robust noise removal model. The model effectively reduces pixel variance caused by environmental factors. This enhanced framework is precious for referred moving object segmentation, where objects identification done based on text queries. By integrating our noise removal module, we ensure better alignment of features, which enhances the precision of referred moving segmentation. Our approach maintains temporal consistency, making object segmentation more reliable under challenging weather conditions while preserving the original video quality by removing weather noise. We have employed separate noise removal modules for haze and rain environmental noise. A ResNet based classifier model trained to identify the noise class on the fly. To demonstrate the effectiveness of our methodology, we selected an ROV benchmark to assess segmentation performance. Experiments on the DAVIS 2017 dataset show that our proposed methodology performs well on weatheraffected videos, significantly improving the benchmark metrics Jaccard (J) and F-measure (F) indices after removing weather noise. Using the benchmark SgMg model for referred segmentation, the mean J&F score is 63.64 without environmental noise. When haze is introduced to the dataset, the mean J&F score drops to 58.71. After applying WARMOS approach, the mean J&F improves to 60.50. A similar pattern is observed for rain: when rain is introduced, the mean J&F score is 61.00, and after applying WARMOS, it improves to 61.06. This highlights our approach's significance in mitigating the impact of environmental noise.