Auto-Curation and Personalization of Sports Highlights Through Multimodal Excitement Measures

Michele Merler, Dhiraj Joshi, Quoc-Bao Nguyen, Stephen Hammer, John Kent, John R. Smith, Rogerio S. Feris; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 1-9

The production of sports highlights packages is an essential task for broadcast media, yet it requires labor-intensive video editing. We propose a novel approach for auto-curation of sports highlights, and use it to create a real-world system for editorial aid of golf highlight reels. Our method fuses information from the player reaction (action recognition such as high-five), spectators (crowd cheering), and commentator (voice tone and word analysis) to determine the most interesting moments of a game. We identify the start and end of key shot highlights with metadata such as player name and hole number, allowing personalized content summarization and retrieval. We also introduce a zero-shot learning framework for our classifiers by exploiting the correlation of different modalities. We demonstrated our system on a major golf tournament.