• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Assistance of multi-media contents generation for supporting learning and teaching of craftsmanship at home

Research Project

Project/Area Number 18K11425
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Review Section Basic Section 61030:Intelligent informatics-related
Research InstitutionThe University of Tokyo

Principal Investigator

Yamakata Yoko  東京大学, 大学院情報理工学系研究科, 准教授 (60423018)

Co-Investigator(Kenkyū-buntansha) 山崎 俊彦  東京大学, 大学院情報理工学系研究科, 教授 (70376599)
今堀 慎治  中央大学, 理工学部, 教授 (90396789)
Project Period (FY) 2018-04-01 – 2023-03-31
Project Status Completed (Fiscal Year 2022)
Budget Amount *help
¥4,420,000 (Direct Cost: ¥3,400,000、Indirect Cost: ¥1,020,000)
Fiscal Year 2020: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Fiscal Year 2019: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Fiscal Year 2018: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Keywords食のAI / 手順文書の意味理解 / 調理支援 / 食事管理アプリケーション / 人間行動認識 / マルチメディア情報処理 / 教示コンテンツ自動生成 / e-learning / ものづくり / レシピ / 料理・調理
Outline of Final Research Achievements

The goal of this research is to enable ordinal people without specialized knowledge to learn and teach “making things” using AI technology. The research leader tackled three challenges on the field of recipe understanding and cooking support task. The first challenge was to develop a natural language processing model for recipe comprehension. We expanded the corpus created in Japanese and English to include Chinese. We also started developing a smartphone app called “RecipeLog” for recording and managing meals based on recipes. Finally, we proposed a method for automatically generating additional explanatory text to improve the readability of recipes. These results were presented at domestic and international conferences and academic journals, including ACM Multimedia 2022.

Academic Significance and Societal Importance of the Research Achievements

調理や裁縫は特別な教育を受けた一部の専門家だけでなく、多くの一般の人々が家庭内で携わっている「ものづくり」であり、その技術やノウハウの共有を支援することに社会的意義がある。本課題では、ものづくりの手順書であるレシピについて、専門教育を受けていない一般の人が記述すると欠損しがちだが、読者の理解を助ける付加的説明文を自動生成することで、その知識共有を支援する技術を開発した。また、レシピを作成する経験に乏しいユーザであっても、AI技術により簡単に自分のレシピを記録できるアプリRecipeLogは、家庭における「ものづくり」の情報収集およびフィールドワークのツールとして活用が見込まれる。

Report

(6 results)
  • 2022 Annual Research Report   Final Research Report ( PDF )
  • 2021 Research-status Report
  • 2020 Research-status Report
  • 2019 Research-status Report
  • 2018 Research-status Report
  • Research Products

    (52 results)

All 2023 2022 2021 2020 2019 2018 Other

All Int'l Joint Research (4 results) Journal Article (13 results) (of which Int'l Joint Research: 1 results,  Peer Reviewed: 4 results,  Open Access: 6 results) Presentation (31 results) (of which Int'l Joint Research: 10 results,  Invited: 1 results) Remarks (4 results)

  • [Int'l Joint Research] University of California, Irvine(米国)

    • Related Report
      2021 Research-status Report
  • [Int'l Joint Research] The University of Sussex(英国)

    • Related Report
      2020 Research-status Report
  • [Int'l Joint Research] University of Sussex(英国)

    • Related Report
      2019 Research-status Report
  • [Int'l Joint Research] The University of Sussex(英国)

    • Related Report
      2018 Research-status Report
  • [Journal Article] Proposal and Evaluation of a Cooking Recipe Writing Support by Generating and Presenting Additional Descriptive Text2022

    • Author(s)
      香川璃奈, 原 悠輔, 姜 志勲, 山肩洋子
    • Journal Title

      電子情報通信学会論文誌D 情報・システム

      Volume: J105-D Issue: 12 Pages: 736-746

    • DOI

      10.14923/transinfj.2022JDP7025

    • ISSN
      1880-4535, 1881-0225
    • Year and Date
      2022-12-01
    • Related Report
      2022 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Recipe Recording by Duplicating and Editing Standard Recipe2022

    • Author(s)
      Akihisa Ishino, Yoko Yamakata, Kiyoharu Aizawa
    • Journal Title

      International Workshop on Multimedia for Cooking, Eating, and related APPlications in conjunction with ACMMM2022 (CEA++22)

      Volume: NA Pages: 37-40

    • DOI

      10.1145/3552485.3554942

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed
  • [Journal Article] MIRecipe: A Recipe Dataset for Stage-Aware Recognition of Changes in Appearance of Ingredients2021

    • Author(s)
      Yixin Zhang, Yoko Yamakata, and Keishi Tajima
    • Journal Title

      ACM Multimedia Asia (MMAsia '21)

      Volume: 16 Pages: 1-1

    • DOI

      10.1145/3469877.3490596

    • Related Report
      2021 Research-status Report
    • Open Access
  • [Journal Article] Noisy Annotation Refinement for Object Detection2021

    • Author(s)
      Jiafeng Mao, Qing Yu, Yoko Yamakata and Kiyoharu Aizawa
    • Journal Title

      British Machine Vision Conference (BMVC)

      Volume: - Pages: 1-1

    • Related Report
      2021 Research-status Report
    • Open Access
  • [Journal Article] Boosting Personalized Food Image Classifier by Sharing Food Records2021

    • Author(s)
      Seum Kim, Yoko Yamakata, and Kiyoharu Aizawa
    • Journal Title

      ACM Multimedia Workshop on CEA (CEA '21)

      Volume: - Pages: 29-29

    • DOI

      10.1145/3463947.3469238

    • Related Report
      2021 Research-status Report
  • [Journal Article] World Food Atlas Project2021

    • Author(s)
      Ali Rostami, Zhouhang Xie, Akihisa Ishino, Yoko Yamakata, Kiyoharu Aizawa, and Ramesh Jain
    • Journal Title

      ACM Multimedia Workshop on CEA (CEA '21)

      Volume: - Pages: 33-33

    • DOI

      10.1145/3463947.3469235

    • Related Report
      2021 Research-status Report
    • Open Access / Int'l Joint Research
  • [Journal Article] RecipeLog: Recipe Authoring App for Accurate Food Recording2021

    • Author(s)
      Akihisa Ishino, Yoko Yamakata, Hiroaki Karasawa and Kiyoharu Aizawa
    • Journal Title

      ACM Multimedia 2021, Technical Demo

      Volume: - Pages: 2798-2798

    • DOI

      10.1145/3474085.3478563

    • Related Report
      2021 Research-status Report
  • [Journal Article] Supplementing Omitted Named Entities in Cooking Procedural Text with Attached Images2021

    • Author(s)
      Yixin Zhang, Yoko Yamakata, Keishi Tajima
    • Journal Title

      IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR2021)

      Volume: - Pages: 199-199

    • DOI

      10.1109/mipr51284.2021.00037

    • Related Report
      2021 Research-status Report
  • [Journal Article] 食行動の自動認識に向けた食事中の一人称視点映像データセットの構築2021

    • Author(s)
      佐藤賢志郎、天野宗佑、山肩洋子、相澤清晴
    • Journal Title

      映像情報メディア学会 メディア工学研究会

      Volume: vol. 46, no. 6 Pages: 19-19

    • Related Report
      2021 Research-status Report
  • [Journal Article] RecipeLog :食事管理のためのスケルトンレシピの作成と応用2021

    • Author(s)
      石野 耀久, 山肩 洋子, 唐澤 弘明, 相澤 清晴
    • Journal Title

      HCGシンポジウム

      Volume: - Pages: 1-1

    • Related Report
      2021 Research-status Report
  • [Journal Article] クックパッドデータセットで学習したBERT及びGPT-2の活用法に関する検討2021

    • Author(s)
      香川璃奈, 原悠輔, 姜志勲,山肩洋子
    • Journal Title

      HCGシンポジウム2021

      Volume: - Pages: 1-1

    • Related Report
      2021 Research-status Report
  • [Journal Article] Structure-Aware Procedural Text Generation From an Image Sequence2020

    • Author(s)
      T. Nishimura, A. Hashimoto, Y. Ushiku, H. Kameko, Y. Yamakata and S. Mori
    • Journal Title

      IEEE Access

      Volume: 9 Pages: 2125-2141

    • DOI

      10.1109/access.2020.3043452

    • Related Report
      2020 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Recognition of Multiple Food Items in A Single Photo for Use in A Buffet-Style Restaurant2019

    • Author(s)
      Masashi ANZAWA, Sosuke AMANO, Yoko YAMAKATA, Keiko MOTONAGA, Akiko KAMEI, Kiyoharu AIZAWA
    • Journal Title

      IEICE Transactions on Information and Systems

      Volume: E102.D Issue: 2 Pages: 410-414

    • DOI

      10.1587/transinf.2018EDL8183

    • NAID

      130007589314

    • ISSN
      0916-8532, 1745-1361
    • Year and Date
      2019-02-01
    • Related Report
      2018 Research-status Report
    • Peer Reviewed / Open Access
  • [Presentation] Food Computing for Human and Planet Health2023

    • Author(s)
      山肩洋子
    • Organizer
      2023 INTERNATIONAL WORKSHOP ON ADVANCED IMAGE TECHNOLOGY (IWAIT)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] RecipeLog:レシピ作成の支援ツール2021

    • Author(s)
      佐藤賢志郎,石野耀久,唐澤弘明,山肩洋子,相澤清晴
    • Organizer
      日本栄養改善学会学術総会
    • Related Report
      2021 Research-status Report
  • [Presentation] 文書の内容と構造が書き手の負担と読み手の効用に与える影響の研究:文書記載支援技術の開発に向けて2021

    • Author(s)
      香川璃奈, 松原正樹, 宮田玲, 馬場雪乃, 山肩洋子
    • Organizer
      2021年度 人工知能学会全国大会論文集 (JSAI2021)
    • Related Report
      2021 Research-status Report
  • [Presentation] 料理レシピの内容と構成が書き手の負担と読み手の効用に与える影響の研究2021

    • Author(s)
      香川璃奈, 松原正樹, 宮田玲, 馬場雪乃, 山肩洋子
    • Organizer
      人工知能学会第26回インタラクティブ情報アクセスと可視化マイニング研究会
    • Related Report
      2020 Research-status Report
  • [Presentation] 既存レシピをテンプレートとするレシピ作成支援アプリRecipeLogの開発2021

    • Author(s)
      石野耀久, 唐澤弘明, 山肩洋子, 相澤清晴
    • Organizer
      データ工学と情報マネジメントに関するフォーラム DEIM Forum 2021
    • Related Report
      2020 Research-status Report
  • [Presentation] Stage-Aware Recognition Method for Foodstuffs Changing in Appearance During Cooking Process on Chinese Recipe2021

    • Author(s)
      Yixin Zhang, Yoko Yamakata, Keishi Tajima
    • Organizer
      データ工学と情報マネジメントに関するフォーラム DEIM Forum 2021
    • Related Report
      2020 Research-status Report
  • [Presentation] English Recipe Flow Graph Corpus2020

    • Author(s)
      Yoko Yamakata, Shinsuke Mori and John Carroll
    • Organizer
      International Conference on Language Resources and Evaluation (LREC)
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] Visual Grounding Annotation of Recipe Flow Graph2020

    • Author(s)
      Taichi Nishimura, Suzushi Tomori, Hayato Hashimoto, Atsushi Hashimoto, Yoko Yamakata, Jun Harashima, Yoshitaka Ushiku and Shinsuke Mori
    • Organizer
      International Conference on Language Resources and Evaluation (LREC)
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] 食事記録分析による長期外出自粛の主観的健康度への影響調査2020

    • Author(s)
      石野耀久,唐澤弘明,天野宗佑,山肩洋子,相澤清晴
    • Organizer
      電子情報通信学会ライフインテリジェンスとオフィス情報システム研究会
    • Related Report
      2020 Research-status Report
  • [Presentation] English Recipe Flow Graph Corpus2020

    • Author(s)
      Yoko Yamakata, Shinsuke Mori and John Carroll
    • Organizer
      International Conference on Language Resources and Evaluation (LREC) 2020
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] Visual Grounding Annotation of Recipe Flow Graph2020

    • Author(s)
      Taichi Nishimura, Suzushi Tomori, Hayato Hashimoto, Atsushi Hashimoto, Yoko Yamakata, Jun Harashima, Yoshitaka Ushiku and Shinsuke Mori
    • Organizer
      International Conference on Language Resources and Evaluation (LREC) 2020
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] 食事記録データからの健康度予測の検討とデータ取得2020

    • Author(s)
      石野耀久, 唐澤弘明, 川原田美雪, 天野宗佑, 江口洋子, 山肩洋子, 相澤清晴
    • Organizer
      第12回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2019 Research-status Report
  • [Presentation] Twitterにおけるトピック間類似度を用いたトピック転換後の人気予測2020

    • Author(s)
      岩永雅史, 田島敬史, 山肩洋子
    • Organizer
      第12回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2019 Research-status Report
  • [Presentation] Complementation of Food and Tool Information in Multi-Modal Recipe Procedural Descriptions2020

    • Author(s)
      Yixin Zhang (京都大学), Yoko Yamakata (東京大学), Keishi Tajima
    • Organizer
      第12回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2019 Research-status Report
  • [Presentation] Frame Selection for Producing Recipe with Pictures from an Execution Video of a Recipe2019

    • Author(s)
      Taichi Nishimura Atsushi Hashimoto, Yoko Yamakata, Shinsuke Mori
    • Organizer
      The 11th Workshop on Multimedia for Cooking and Eating Activities
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] Categorization of Cooking Actions Based on Textual/Visual Similarity2019

    • Author(s)
      Yixing Zhang, Yoko Yamakata, Keishi Tajima
    • Organizer
      Proc. of 5th International Workshop on Multimedia Assisted Dietary Management (MADiMa) (In conj. with ACMMM)
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] 大規模レシピデータとテキストベースの食事記録に基づくアスリートの食傾向分析2019

    • Author(s)
      川原田美雪・唐澤弘明・坂本美沙子・天野宗佑・山肩洋子・相澤清晴
    • Organizer
      電子情報通信学会 ライフインテリジェンスとオフィス情報システム研究会 (LOIS)
    • Related Report
      2019 Research-status Report
  • [Presentation] 大規模レシピに基づく食事記録のカテゴリ推定による食物摂取頻度解析:アスリートを対象 にした実践2019

    • Author(s)
      川原田美雪,唐澤弘明,坂本美沙子,天野宗祐,山肩洋子,相澤清晴
    • Organizer
      情報科学技術フォーラム、FIT2019, O-038
    • Related Report
      2019 Research-status Report
  • [Presentation] 食事名とレシピデータとのマッチングによるアスリートの食傾向分析2019

    • Author(s)
      川原田美雪,唐澤弘明,坂本美沙子,天野宗祐,山肩洋子,相澤清晴
    • Organizer
      日本スポーツ栄養学会 第6回大会, 3-O-16-2
    • Related Report
      2019 Research-status Report
  • [Presentation] "質問回答サービスにおける人から人への質問拡散行為を用いた質問提示システムの評価"2019

    • Author(s)
      中川瑛慎, 田島敬史, 山肩洋子
    • Organizer
      第11回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "ユーザとトピックを指定するTwitter検索における最適な擬似適合フィードバック文書数の動的決定"2019

    • Author(s)
      片岡大祐, 田島敬史, 山肩洋子
    • Organizer
      第11回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "Find Target Data Fast! A Method and Its Behavior of Target Data Collection for Online Annotation using Machine Learning Method"2019

    • Author(s)
      李瑞徳,山肩洋子, 田島敬史
    • Organizer
      第11回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "写真付き手順書生成のための実施映像からのフレーム選択"2019

    • Author(s)
      西村太一, 橋本敦史, 山肩洋子, 森信介
    • Organizer
      第11回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "非母国語検索者向けの機械翻訳を活用したweb検索結果提示手法の提案"2019

    • Author(s)
      佐々岡哲哉, 山肩洋子, 田島敬史
    • Organizer
      第11回データ工学と情報マネジメントに関するフォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "大規模食事記録データと深層学習モデルによる食習慣ベクトル化手法の検討"2019

    • Author(s)
      合田悠治, 天野宗佑, 山肩洋子, 相澤清晴
    • Organizer
      映像情報メディア学会 メディア工学研究会
    • Related Report
      2018 Research-status Report
  • [Presentation] "アスリートの食事管理を支援するフードコンサルティングシステム"2019

    • Author(s)
      唐澤弘明, 川原田美雪, 天野宗佑, 山肩洋子, 相澤清晴
    • Organizer
      電子情報通信学会 総合大会
    • Related Report
      2018 Research-status Report
  • [Presentation] "大規模食事記録データの栄養価クラスタリングに基づく食習慣ベクトルBag-of-Foodsとその検証"2018

    • Author(s)
      合田悠治, 天野宗祐, 山肩洋子, 相澤清晴
    • Organizer
      情報科学技術フォーラム
    • Related Report
      2018 Research-status Report
  • [Presentation] "管理栄養士とアスリートのためのフードコンサルティングシステム"2018

    • Author(s)
      唐澤弘明, 川原田美雪, 山肩洋子, 相澤清晴
    • Organizer
      映像情報メディア学会冬季大会 2018
    • Related Report
      2018 Research-status Report
  • [Presentation] "A Case Study on Start-up of Dataset Construction: In Case of Recipe Named Entity Corpus"2018

    • Author(s)
      Yoko Yamakata, Keishi Tajima, and Shinsuke Mori
    • Organizer
      The Second IEEE Workshop on Human-in-the-loop Methods and Human Machine Collaboration in BigData (IEEE HMData2018)"
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Presentation] "Bag-of-foods: analysis of personal foodlogging data"2018

    • Author(s)
      Yuji Goda, Sosuke Amano, Yoko Yamakata, Kiyoharu Aizawa
    • Organizer
      IJCAI Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Presentation] "Resource intensity for menu items: how much land is required to provide for each dish?"2018

    • Author(s)
      Yoko Yamakata, Eiji Yamasue, Benjamin McLellan, Kiyoharu Aizawa
    • Organizer
      IJCAI Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Remarks] App StoreにおけるRecipeLogのインストールサイト

    • URL

      https://apps.apple.com/us/app/recipelog/id1587480365

    • Related Report
      2021 Research-status Report
  • [Remarks] Google PlayにおけるRecipe Logのインストールサイト

    • URL

      https://play.google.com/store/apps/details?id=recipe.log.app

    • Related Report
      2021 Research-status Report
  • [Remarks] 東京大学 情報理工学系研究科 山肩研究室 研究リソース

    • URL

      https://sites.google.com/view/yy-lab/resource

    • Related Report
      2020 Research-status Report 2019 Research-status Report
  • [Remarks] 京都大学 学術情報メディアセンター 森研究室 レシピ言語処理マニュアル

    • URL

      http://www.ar.media.kyoto-u.ac.jp/how-to/recipe-NLP/

    • Related Report
      2019 Research-status Report

URL: 

Published: 2018-04-23   Modified: 2024-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi