Physics-Enhanced Machine Learning for Virtual Fluorescence Microscopy

Colin L. Cooke, Fanjie Kong, Amey Chaware, Kevin C. Zhou, Kanghyun Kim, Rong Xu, D. Michael Ando, Samuel J. Yang, Pavan Chandra Konda, Roarke Horstmeyer; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 3803-3813


This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. We use a deep neural network (DNN) to effectively design optical patterns for specimen illumination that substantially improve upon the ability to infer fluorescence image information from unstained microscope images. To achieve this design, we include an illumination model within the DNN's first layers that is jointly optimized during network training. We validated our method on two different experimental setups, with different magnifications and sample types, to show a consistent improvement in performance as compared to conventional microscope imaging methods. Additionally, to understand the importance of learned illumination on the inference task, we varied the number of illumination patterns being optimized (and thus the number of unique images captured) and analyzed how the structure of the patterns changed as their number increased. This work demonstrates the power of programmable optical elements at enabling better machine learning algorithm performance and at providing physical insight into next generation of machine-controlled imaging systems.

Related Material

[pdf] [supp] [arXiv]
@InProceedings{Cooke_2021_ICCV, author = {Cooke, Colin L. and Kong, Fanjie and Chaware, Amey and Zhou, Kevin C. and Kim, Kanghyun and Xu, Rong and Ando, D. Michael and Yang, Samuel J. and Konda, Pavan Chandra and Horstmeyer, Roarke}, title = {Physics-Enhanced Machine Learning for Virtual Fluorescence Microscopy}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {3803-3813} }