Compressive Light Field Reconstructions Using Deep Learning

Mayank Gupta, Arjun Jauhari, Kuldeep Kulkarni, Suren Jayasuriya, Alyosha Molnar, Pavan Turaga; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 11-20

Single-shot light field cameras typically sacrifice spatial resolution to sample angular viewpoints, multiplexing rays onto a 2D sensor array. Using compressive sensing to recover this resolution requires long processing times for iterative solvers. We present a new, two branch neural network architecture to efficiently recover a high resolution 4D light field from a single coded 2D image. This network achieves average PSNR values of 26-32 dB on simulated data, outperforming the baseline of dictionary-based learning. In addition, reconstruction time is decreased from 35 minutes to 6.7 minutes compared to the dictionary method for equivalent visual quality at small sampling/compression ratios as low as 8%. We test our network reconstructions on synthetic light fields, simulated coded measurements of real Lytro Illum light fields, and real coded data from a custom CMOS diffractive light field camera to show the potential for real-time light field video systems in the future.