Event-based Tracking and Imaging of Randomly Moving Objects in Dense Dynamical Scattering Media

Ning Zhang, Timothy Shea, Arto Nurmikko; Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, 2025, pp. 5114-5125

Acquiring high resolution images of objects of interest which are entirely obscured by dense fog or by any dense scattering media is a perennial challenge for many applications. The problem of simultaneously both tracking and imaging of randomly moving objects presents a particularly vexing challenge. In this work, we develop an end-to-end neuromorphic optical engineering and computational approach to show how to track and image invisible dynamical objects by combining a dynamic vision sensor (DVS) camera with a multistage neuromorphic deep learning approach. Here the novelty in employing a DVS camera lies in part in its inherent capability to isolate dynamical features of objects so that the dominant but uninformative background scattered light is effectively subtracted. Then, in our deep spiking neural network (SNN) model the dual function of tracking and image reconstruction is performed by parallel intertwined modules running in discrete time steps over the event duration. The human vision inspired scheme is tested on the benchtop where randomly moving characters from standardized sets were projected as structured light through a fog chamber with variable optical thickness of up to 4.5. While unrecognizable to the human eye, the system was able to reconstruct images and track the dynamical objects with high accuracy (SSIM: 0.9340; MSE: 0.0178). The results highlight the advantages of a holistic neuromorphic approach in achieving high computational efficiency at low latency and low power consumption to image dynamic objects in changing turbid media.