Predicting Sentiments in Image Advertisements Using Semantic Relations Among Sentiment Labels

Stephen Pilli, Manasi Patwardhan, Niranjan Pedanekar, Shirish Karande; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020, pp. 408-409

Understanding the sentiments evoked by advertisements is crucial in serving them appropriately to consumers. Advertisements often use images to evoke sentiments. An image can convey multiple sentiments of different nature. Automatically predicting these multiple sentiments can help serve better advertisements to consumers, especially in an online scenario at scale. In this paper, we present a neural network model based on graph convolution to predict such sentiments, which exploits the semantic relationship among the sentiment labels. We use it to predict multiple sentiment labels using an annotated dataset of 30,340 image-based advertisements. We also find a distance metric that best represents the distribution of sentiments in the dataset and utilizes it in a loss function that separates applicable sentiments from the non-applicable ones. We report an improvement in mean average precision and overall F1 score over a multi-modal multi-task state-of-the-art model.