Multi-View Multi-Modal Feature Embedding for Endomicroscopy Mosaic Classification

Yun Gu, Jie Yang, Guang-Zhong Yang; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 11-19

Probe-based confocal laser endomicroscopy (pCLE) is an emerging tool for epithelial cancer diagnosis, which enables in vivo microscopic imaging during endoscopic procedures. As a new technique, definite clinical diagnosis is still referenced to the gold standard histology images. In this paper, we propose a Multi-View Multi-Modal Embedding framework (MVMME) to learn representative features for pCLE videos exploiting both pCLE mosaic and histology images. Each pCLE mosaic is represented by multiple feature representations including SIFT, Texton and HoG. A latent space is discovered by embedding the visual features from both mosaics and histology images in a supervised scheme. The features extracted from the latent spaces can make use of multi-modal imaging sources that are more discriminative than unimodal features from mosaics alone. The experiments based on real pCLE datasets demonstrate that our approach outperforms, with statistical significance, several single-view or single-modal methods. A binary classification accuracy of 96% has been achieved.