LSTM Self-Supervision for Detailed Behavior Analysis

Biagio Brattoli, Uta Buchler, Anna-Sophia Wahl, Martin E. Schwab, Bjorn Ommer; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6466-6475

Abstract


Behavior analysis provides a crucial non-invasive and easily accessible diagnostic tool for biomedical research. A detailed analysis of posture changes during skilled motor tasks can reveal distinct functional deficits and their restoration during recovery. Our specific scenario is based on a neuroscientific study of rodents recovering from a large sensorimotor cortex stroke and skilled forelimb grasping is being recorded. Given large amounts of unlabeled videos that are recorded during such long-term studies, we seek an approach that captures fine-grained details of posture and its change during rehabilitation without costly manual supervision. Therefore, we utilize self-supervision to automatically learn accurate posture and behavior representations for analyzing motor function. Learning our model depends on the following fundamental elements: (i) limb detection based on a fully convolutional network is ini- tialized solely using motion information, (ii) a novel self- supervised training of LSTMs using only temporal permu- tation yields a detailed representation of behavior, and (iii) back-propagation of this sequence representation also im- proves the description of individual postures. We establish a novel test dataset with expert annotations for evaluation of fine-grained behavior analysis. Moreover, we demonstrate the generality of our approach by successfully applying it to self-supervised learning of human posture on two standard benchmark datasets.

Related Material


[pdf] [poster]
[bibtex]
@InProceedings{Brattoli_2017_CVPR,
author = {Brattoli, Biagio and Buchler, Uta and Wahl, Anna-Sophia and Schwab, Martin E. and Ommer, Bjorn},
title = {LSTM Self-Supervision for Detailed Behavior Analysis},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {July},
year = {2017}
}