Semantic Video CNNs Through Representation Warping

Raghudeep Gadde, Varun Jampani, Peter V. Gehler; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4453-4462

Abstract


In this work, we propose a technique to convert CNN models for semantic segmentation of static images into CNNs for video data. We describe a warping method that can be used to augment existing architectures with very little extra computational cost. This module is called NetWarp and we demonstrate its use for a range of network architectures. The main design principle is to use optical flow of adjacent frames for warping internal network representations across time. A key insight of this work is that fast optical flow methods can be combined with many different CNN architectures for improved performance and end-to-end training. Experiments validate that the proposed approach incurs only little extra computational cost, while improving performance, when video streams are available. We achieve new state-of-the-art results on the CamVid and Cityscapes benchmark datasets and show consistent improvements over different baseline networks. Our code and models are available at http://segmentation.is.tue.mpg.de

Related Material


[pdf] [supp] [arXiv] [video]
[bibtex]
@InProceedings{Gadde_2017_ICCV,
author = {Gadde, Raghudeep and Jampani, Varun and Gehler, Peter V.},
title = {Semantic Video CNNs Through Representation Warping},
booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
month = {Oct},
year = {2017}
}