Fast Trajectory End-Point Prediction With Event Cameras for Reactive Robot Control

Marco Monforte, Luna Gava, Massimiliano Iacono, Arren Glover, Chiara Bartolozzi; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023, pp. 4036-4044

Prediction can be crucial for tasks with short time constraints if a robot has limited speed and power. A low-latency, high-frequency perception system can reduce the time needed to converge on the expectation of the future state of the world, giving the robot additional time to act - or to choose a safer action. In this paper, we exploit event cameras for asynchronous motion-driven sampling, inherent data compression, and sub-millisecond latency to reduce the convergence time of a data-driven trajectory prediction algorithm. As a use-case, we use a Panda robotic arm to intercept a ball bouncing on a table. To predict the interception point as early as possible, and cope with the intrinsic variability of trajectory length - that cannot be defined a-priori for event cameras - we adopt a Stateful Long Short-Term Memory, that asynchronously updates the prediction for each incoming point of the trajectory and does not require a predefined, fixed length input. We adopt a sim-to-real methodology in which the network is first trained on simulated data and then fine-tuned on real trajectories. Experimental results demonstrate that the dense spatial sampling performed by event cameras significantly increases the number of intercepted trajectories compared to a fixed temporal sampling typical of traditional "frame-based" cameras. Results motivate further exploration of the use of event cameras for prediction in higher-complexity robotic tasks.