Unsupervised Generative Approach for Anomaly Detection to Enhance the Quality of Unseen Medical Datasets

Zhemin Zhang, Bhavika Patel, Bhavik Patel, Imon Banerjee; Proceedings of the Winter Conference on Applications of Computer Vision (WACV) Workshops, 2025, pp. 287-296

Out-of-distribution (OOD) and anomaly detection are critical for reducing noise and improving the generalization of AI models in breast cancer screening especially when handling unseen private data. Given the challenge of limited prior knowledge about OOD samples in external unseen datasets unsupervised generative learning is a preferable solution which trains the model to discern the normal characteristics of in-distribution (ID) data. The hypothesis is that during inference the model aims to reconstruct ID samples accurately while OOD samples exhibit poorer reconstruction due to their divergence from normality. Inspired by state-of-the-art (SOTA) hybrid architectures combining CNNs and transformers we developed a novel backbone -- HAND (Hybrid ANomaly Detection) for detecting OOD from digital mammogram studies. To boost learning efficiency we incorporated synthetic OOD samples and a parallel discriminator in the latent space to distinguish between ID and OOD samples. Gradient reversal to the OOD reconstruction loss penalizes the model for learning OOD reconstructions. An anomaly score is computed by weighting the reconstruction and the discriminator loss. On internal held-out test and external dataset the proposed HAND model outperformed encoder-based and GAN-based baselines and interestingly it also outperformed the hybrid CNN+transformer baselines. Proposed HAND pipeline offers an automated and efficient computational solution for domain-specific quality checks in external screening mammograms yielding actionable insights without direct exposure to private medical imaging data.