1
00:00:00,000 --> 00:00:10,000
We present our work entitled “Panoramas from Photons” which enables the creation of high-quality single photon planar reconstructions under fast motion and low light. 

2
00:00:10,000 --> 00:00:15,000
Motion estimation from a sequence of images is a fundamental problem in computer vision.

3
00:00:15,000 --> 00:00:19,000
However, traditional methods are limited by what the camera can recover.

4
00:00:19,000 --> 00:00:29,000
In low light, a conventional camera will compensate by either increasing the exposure, thereby introducing motion blur, or increasing the gain and introducing noise.

5
00:00:29,000 --> 00:00:40,000
Here we see that for images with blur and noise, image registration -- which is a common method used to estimate motion from images -- does not work reliably.

6
00:00:40,000 --> 00:00:46,000
A possible way to overcome this issue is to use dedicated high-speed cameras.

7
00:00:46,000 --> 00:00:53,000
However, such cameras are not only bulky and costly but also suffer from an extremely low signal-to-noise ratio.

8
00:00:54,000 --> 00:01:00,000
In this work, we instead focus on an emerging class of sensors called single-photon cameras (SPCs). 

9
00:01:00,000 --> 00:01:05,000
These single-photon cameras are capable of high-speed imaging in low-light conditions. 

10
00:01:05,000 --> 00:01:10,000
Our prototype, shown here, can capture 100,000 frames per second.

11
00:01:10,000 --> 00:01:12,000
But there's a catch.

12
00:01:12,000 --> 00:01:15,000
Each captured frame is binary-valued.

13
00:01:15,000 --> 00:01:22,000
A pixel is on if at least one photon is detected during the exposure time and off otherwise.

14
00:01:22,000 --> 00:01:26,000
In the middle, you can see the raw frames captured by our setup.

15
00:01:26,000 --> 00:01:33,000
Notice that to show these binary frames, we had to slow down the video by a huge factor, and it still appears fast.

16
00:01:33,000 --> 00:01:44,000
On the right, we can see that any motion will cause a naive reconstruction, created by simply averaging neighboring frames, to be extremely blurry.

17
00:01:45,000 --> 00:01:57,000
In this work, we propose a stratified reconstruction technique capable of estimating rapid motion from a sequence of high-speed binary frames captured by such a single-photon camera.

18
00:01:57,000 --> 00:02:07,000
Once the motion is properly estimated, we can combine all binary frames into an image that is larger than the sensor's field of view, as shown here on the right.

19
00:02:07,000 --> 00:02:09,000
Here's how it works.

20
00:02:09,000 --> 00:02:17,000
We first capture a sequence of high-speed binary frames along some motion trajectory, represented here as the dotted black line.

21
00:02:17,000 --> 00:02:21,000
Think of each individual dash as a binary frame.

22
00:02:21,000 --> 00:02:29,000
If we knew this motion trajectory, we could compensate for it and perfectly reconstruct the scene from this sequence of binary frames.

23
00:02:29,000 --> 00:02:35,000
However, the motion trajectory is originally unknown to us, as shown here by the blue line.

24
00:02:36,000 --> 00:02:46,000
We start off by selecting contiguous groups of binary frames and naively aggregating them together to create the blurry images you see here.

25
00:02:46,000 --> 00:02:54,000
We then use an off-the-shelf registration method to localize these frames and give us a rough estimate of the motion trajectory.

26
00:02:54,000 --> 00:03:01,000
Due to the blur, the estimate might be very noisy, as shown here by the black lines.

27
00:03:01,000 --> 00:03:06,000
Nevertheless, we can use these noisy estimates to update our trajectory.

28
00:03:06,000 --> 00:03:16,000
And, at the next iteration, we select new groups of binary frames to merge together and use the updated motion estimate to warp all binary frames.

29
00:03:16,000 --> 00:03:22,000
The resulting aggregate images are less blurry and will enable better subsequent localization.

30
00:03:22,000 --> 00:03:30,000
This process is repeated until we've converged, which, in practice, only takes a handful of iterations.

31
00:03:30,000 --> 00:03:36,000
For instance, here is a panoramic reconstruction after one iteration of our pipeline.

32
00:03:36,000 --> 00:03:43,000
Notice that there remains strong noise and distortions due to the incorrect motion estimate.

33
00:03:43,000 --> 00:03:48,000
After a second iteration, some distortions persist.

34
00:03:48,000 --> 00:03:54,000
And finally, after three iterations of our pipeline, we have a straightened panorama with little to no noise.

35
00:03:54,000 --> 00:04:00,000
As you can see, any registration errors or drift that occurred early on gets progressively corrected,

36
00:04:00,000 --> 00:04:05,000
leading to globally consistent matching.

37
00:04:05,000 --> 00:04:10,000
We can also create large-scale panoramas under fast motion.

38
00:04:10,000 --> 00:04:15,000
On the left, we show a video reconstruction of a scene played back in real-time.

39
00:04:15,000 --> 00:04:18,000
Here it is again slowed down by a factor of four.

40
00:04:18,000 --> 00:04:26,000
The panorama on the right was created with roughly 55,000 binary frames and captured in less than half a second.

41
00:04:26,000 --> 00:04:32,000
Contrast that to the time it takes to capture a panorama on your cell phone.

42
00:04:32,000 --> 00:04:35,000
We can also stabilize high-frequency camera shake.

43
00:04:35,000 --> 00:04:41,000
Again, we show the slowed-down video reconstruction here to give a sense of just how fast this motion was.

44
00:04:41,000 --> 00:04:46,000
Super-resolution is also possible thanks to sub-pixel alignment.

45
00:04:46,000 --> 00:04:52,000
The high-frequency motion trajectory, shown here in red, is also recovered.

46
00:04:52,000 --> 00:04:57,000
Finally, I want to show a quick example of when this method fails.

47
00:04:57,000 --> 00:05:03,000
Here we have a sequence shot at night and illuminated exclusively by artificial lights.

48
00:05:03,000 --> 00:05:11,000
Notice that our hardware prototype is fast enough to detect and even measure the flickering of artificial lights due to the electric grid.

49
00:05:11,000 --> 00:05:23,000
While this can be useful in its own right, it causes the underlying registration algorithm to fail as a key assumption of feature-based matching, namely brightness constancy, is violated.

50
00:05:23,000 --> 00:05:27,000
For more results, please see our paper. Thank you.

