Live Demonstration: Joint Estimation of Optical Flow and Intensity Image From Event Sensors

Prasan Shedligeri, Kaushik Mitra; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019, pp. 0-0

Event sensors provide asynchronous, high-temporal rate information about the pixel-level brightness changes in the scene. Temporal information is lost when these event sensor data is converted into an event frame. This temporal information can be recovered if these event frames are processed as a sequence. We propose a deep learning based method to reconstruct high-quality, high-dynamic range, high frame rate and temporally consistent pseudo-images which can run in real-time. Our proposed method can reconstruct pseudo-images at high temporal resolution, even though it is supervised using intensity images from a low-frame rate sensor. We propose convolutional-LSTM based seq2seq deep learning model which takes in as input a sequence of event frames and reconstructs a sequence of pseudo-images. To further enhance the quality of our reconstructed pseudo-images, we propose a model to jointly learn to reconstruct pseudo-images and optical flow. We propose a novel brightness agnostic loss function to supervise the training of pseudo-images. The model learns to estimate optical flow using a self-supervised learning method. We show that our model can produce updates at upto 150 Hz on a GPU while out-performing previous state-of-the-art methods in reconstruction quality. We quantitatively show that our joint learning model to estimate optical flow performs comparably with previous state-of-the-art methods which are tuned only to estimate optical flow. We show that joint estimation of optical flow and pseudo-images leads to better reconstruction quality of the pseudo-images.