Temporal Attention Mechanism with Conditional Inference for Large-Scale Multi-Label Video Classification

Eun-Sol Kim, Kyoung-Woon On, Jongseok Kim, Yu-Jung Heo, Seong-Ho Choi, Hyun-Dong Lee, Byoung-Tak Zhang; Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0-0

Abstract


Here we show neural network based methods, which combine multimodal sequential inputs effectively and classify the inputs into multiple categories. Two key ideas are 1) to select informative frames among a sequence using attention mechanism and 2) to utilize correlation information between labels to solve multi-label classification problems. The attention mechanism is used in both modality (spatio) and sequential (temporal) dimensions to ignore noisy and meaningless frames. Furthermore, to tackle fundamental problems induced by independently predicting each label in conventional multi-label classification methods, the proposed method considers the dependencies among the labels by decomposing joint probability of labels into conditional terms. From the experimental results (5th in the Kaggle competition), we discuss how the suggested methods operate in the YouTube-8M Classification Task, what insights they have, and why they succeed or fail.

Related Material


[pdf]
[bibtex]
@InProceedings{Kim_2018_ECCV_Workshops,
author = {Kim, Eun-Sol and On, Kyoung-Woon and Kim, Jongseok and Heo, Yu-Jung and Choi, Seong-Ho and Lee, Hyun-Dong and Zhang, Byoung-Tak},
title = {Temporal Attention Mechanism with Conditional Inference for Large-Scale Multi-Label Video Classification},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops},
month = {September},
year = {2018}
}