Food Portion Estimation via 3D Object Scaling

Gautham Vinod, Jiangpeng He, Zeman Shao, Fengqing Zhu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 3741-3749

Image-based methods to analyze food images have alleviated the user burden and biases associated with traditional methods. However accurate portion estimation remains a major challenge due to the loss of 3D information in the 2D representation of foods captured by smartphone cameras or wearable devices. In this paper we propose a new framework to estimate both food volume and energy from 2D images by leveraging the power of 3D food models and physical reference in the eating scene. Our method estimates the pose of the camera and the food object in the input image and recreates the eating occasion by rendering an image of a 3D model of the food with the estimated poses. We also introduce a new dataset SimpleFood40 which contains 2D images of 40 food items and associated annotations including food volume weight and energy. Our method achieves an average error of 31.10 kCal (17.67%) on this dataset outperforming existing portion estimation methods.