Im2Calories: Towards an Automated Mobile Vision Food Diary

Austin Meyers, Nick Johnston, Vivek Rathod, Anoop Korattikara, Alex Gorban, Nathan Silberman, Sergio Guadarrama, George Papandreou, Jonathan Huang, Kevin P. Murphy; The IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1233-1241


We present a system which can recognize the contents of your meal from a single image, and then predict its nutritional contents, such as calories. The simplest version assumes that the user is eating at a restaurant for which we know the menu. In this case, we can collect images offline to train a multi-label classifier. At run time, we apply the classifier (running on your phone) to predict which foods are present in your meal, and we lookup the corresponding nutritional facts. We apply this method to a new dataset of images from 23 different restaurants, using a CNN-based classifier, significantly outperforming previous work. The more challenging setting works outside of restaurants. In this case, we need to estimate the size of the foods, as well as their labels. This requires solving segmentation and depth / volume estimation from a single image. We present CNN-based approaches to these problems, with promising preliminary results.

Related Material

author = {Meyers, Austin and Johnston, Nick and Rathod, Vivek and Korattikara, Anoop and Gorban, Alex and Silberman, Nathan and Guadarrama, Sergio and Papandreou, George and Huang, Jonathan and Murphy, Kevin P.},
title = {Im2Calories: Towards an Automated Mobile Vision Food Diary},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {December},
year = {2015}