CS-UNet: A Flexible Segmentation Algorithm for Microscopy Images

Khaled Alrfou, Tian Zhao, Amir Kordijaz; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8114-8119

CS-UNet is a U-shaped image-segmentation algorithm with parallel CNN and Transformer encoders. This algorithm leverages the relative strength of CNN and Transformers and enables flexible combination of encoders pre-trained on different datasets to extract the maximum benefit of transfer-learning. CS-UNet is evaluated for its segmentation accuracy on microscopy images of materials science. The performance of CS-UNet is comparable or better than state-of-the-art algorithms based on CNN or Transformer encoders. Pre-training the encoders of CS-UNet on microscopy images further improves its performance in out-of-distribution learning and one-shot learning. The Intersection over Union (IoU) accuracy of nickel-based super-alloy images is improved from 77.89% to 82.13% for out-of-distribution learning and IoU accuracy of environmental-barrier-coating images is improved from 65.9% to 70.45% for one-shot learning. This suggests that Transformer and CNN complement each other and pre-training on images with similar attributes is beneficial to the downstream tasks.