4D Effect Video Classification With Shot-Aware Frame Selection and Deep Neural Networks

Thomhert S. Siadari, Mikyong Han, Hyunjin Yoon; Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1148-1155

A 4D effect video played at cinema or other designated places is a video annotated with physical effects such as motion, vibration, wind, flashlight, water spray, and scent. In order to automate the time-consuming and labor-intensive process of creating such videos, we propose a new method to classify videos into 4D effect types with shot-aware frame selection and deep neural networks (DNNs). Shot-aware frame selection is a process of selecting video frames across multiple shots based on the shot length ratios to subsample every video down to a fixed number of frames for classification. For empirical evaluation, we collect a new dataset of 4D effect videos where most of the videos consist of multiple shots. Our extensive experiments show that the proposed method consistently outperforms DNNs without considering multi-shot aspect by up to 8.8% in terms of mean average precision.