Automatic Food Calorie Estimation from Photos Employing Deep Learning and Food-related Knowledge on the Web

2020 Fiscal Year Final Research Report

• Project/Area Number: 17H01745
• Research Category: Grant-in-Aid for Scientific Research (B)
• Allocation Type: Single-year Grants
• Section: 一般
• Research Field: Multimedia database
• Research Institution: The University of Electro-Communications
• Principal Investigator: Yanai Keiji 電気通信大学, 大学院情報理工学研究科, 教授 (20301179)
• Co-Investigator(Kenkyū-buntansha): 大河原 一憲 電気通信大学, 大学院情報理工学研究科, 准教授 (30631270)
• Project Period (FY): 2017-04-01 – 2021-03-31
• Keywords: 食事画像認識 / 食事画像変換 / 深層学習 / 食事AR

Outline of Final Research Achievements

In this study, we mainly studied the following five points to achieve highly accurate estimation of the amount of calories and nutrients in meals from photographs by using Web big data and deep learning. (1) Multi-task CNN-based calorie estimation for single-item meal images. (2) Individual meal calorie estimation for multiple meal images. (3) Realization of meal detection using region segmentation and rectangles, and creation of datasets for this purpose. (3) 3D meal shape estimation. We implemented 3D meal shape estimation for more accurate meal volume estimation. (4) High-Resolution food image translation using a large-scale Web food images and its application to food AR. (5) A new weakly-supervised region segmentation method is proposed and applied into food domain.

Free Research Field

画像認識，深層学習

Academic Significance and Societal Importance of the Research Achievements

本研究は，画像認識技術および深層学習技術を用いた画像ベースの食事画像分析技術の実用化を目指した高精度化を目的としており，そのための技術開発および学習データセットの構築を行った．個々の技術開発に加えて，現状，ほとんど公開データが存在していない，食事の領域分割データセットや3Dモデル食事データセットを構築・公開することで，世界の研究コミュニティに対しても貢献を行っている．