Refining Biologically Inconsistent Segmentation Masks with Masked Autoencoders

Alexander Sauer, Yuan Tian, Joerg Bewersdorf, Jens Rittscher; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 6904-6912

Microscopy images often feature regions of low signal-to-noise ratio (SNR) which leads to a considerable amount of ambiguity in the correct corresponding segmentation. This ambiguity can introduce inconsistencies in the segmentation mask which violate known biological constraints. In this work we present a methodology which identifies areas of low SNR and refines the segmentation masks such that they are consistent with biological structures. Low SNR regions with uncertain segmentation are detected using model ensembling and selectively restored by a masked autoencoder (MAE) which leverages information about well-imaged surrounding areas. The prior knowledge of biologically consistent segmentation masks is directly learned from the data. We validate our approach in the context of analysing intracellular structures specifically by refining segmentation masks of mitochondria in expansion microscopy images with a global staining.