Learning Latent Temporal Connectionism of Deep Residual Visual Abstractions for Identifying Surgical Tools in Laparoscopy Procedures

Kaustuv Mishra, Rachana Sathish, Debdoot Sheet; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 58-65

Surgical workflow in minimally invasive interventions like laparoscopy can be modeled with the aid of tool usage information. The video recording during the surgery primarily for viewing the surgical site using endoscope can be leveraged for this purpose without the need for additional sensors or instruments. We propose a method which learns to detect the tool presence in laparoscopy videos by leveraging the temporal information of the systematically executed surgical procedures and higher abstractions of the spatial visual features extracted from the surgical video. We propose a framework consisting of using Convolutional Neural Networks for extracting the visual features and Long Short-Term Memory network to encode the temporal information. The proposed framework has been experimentally verified using a publicly available dataset consisting of 10 training and 5 testing annotated videos with an average accuracy of 88.75% in detection of tool presence.