• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Organizing Videos by Human-in-the-Loop Machine Learning for for Skill Transfer and Know-How Sharing

Research Project

Project/Area Number 20K12115
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Review Section Basic Section 62030:Learning support system-related
Research InstitutionTokyo University of Science

Principal Investigator

Taniguchi Yukinobu  東京理科大学, 工学部情報工学科, 教授 (70759422)

Co-Investigator(Kenkyū-buntansha) 古田 諒佑  東京大学, 生産技術研究所, 助教 (20843535)
Project Period (FY) 2020-04-01 – 2023-03-31
Project Status Completed (Fiscal Year 2022)
Budget Amount *help
¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2022: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Fiscal Year 2021: ¥1,170,000 (Direct Cost: ¥900,000、Indirect Cost: ¥270,000)
Fiscal Year 2020: ¥1,950,000 (Direct Cost: ¥1,500,000、Indirect Cost: ¥450,000)
Keywords人間参加型機械学習 / Human-in-the-loop / 弱教師あり学習 / 技能伝承 / 画像認識 / 機械学習 / 人間参加型
Outline of Research at the Start

製造業・看護など様々な業界において,熟練者から初心者への技能伝承,組織内でのノウハウ共有が重要な課題となっている.言語化が難しい技能やノウハウを共有する手段として「映像」が有効であるが,組織内で映像を共有する前に最低限の編集(映像のシーンにコメントや注釈を付与する作業)は必要であり,素人にとって心理的ハードルが高いという問題がある.本研究計画では,教師データ作成に多大なコストをかけることなく高精度な画像認識モデルを構築するために,人と機械が協調してシステムを賢くしていく人間参加型(Human-in-the-loop)機械学習を採用した新たな学習手法を創出する.

Outline of Final Research Achievements

This research aims to promote skill transfer and know-how sharing by automatically organizing unedited work videos --- dividing them into semantic scenes and tagging them --- using image recognition. To reduce the annotation burden associated with the development of image recognition models, we investigated the following two approaches: (1) weakly supervised and transfer learning approach, which trains image recognition models by taking advantage of the small amount of incomplete annotation data provided by the user, and (2) a human-in-the-loop machine learning approach, which improves image recognition models by presenting the image recognition results to the user and obtaining the user feedback.

Academic Significance and Societal Importance of the Research Achievements

製造業・看護など様々な業界において,熟練者から初心者への技能伝承,組織内でのノウハウ共有が重要な課題となっている.技能伝承・ノウハウは,企業やコミュニティに閉じたドメイン固有の概念を扱うため,学習データの整備,汎用的な画像認識モデルの構築が難しいことが問題となっている.本研究成果は,この問題点の解決に向けたもので,技能伝承・ノウハウ共有の促進に資するものである.なお,ここで開発した手法は,学習データ整備の困難を軽減するもので,当初想定した分野以外への展開も可能であることを示した.

Report

(4 results)
  • 2022 Annual Research Report   Final Research Report ( PDF )
  • 2021 Research-status Report
  • 2020 Research-status Report
  • Research Products

    (16 results)

All 2023 2022 2021 2020 Other

All Journal Article (4 results) (of which Peer Reviewed: 4 results,  Open Access: 3 results) Presentation (10 results) (of which Int'l Joint Research: 2 results) Remarks (2 results)

  • [Journal Article] Detecting Mounting Behaviors of Dairy Cows by Pre-Training with Pseudo Images2023

    • Author(s)
      Y. Okuda, Y. Yamamoto, K. Nakamura and Y. Taniguchi
    • Journal Title

      Proceedings the 29th International Workshop on Frontiers of Computer Vision (IW-FCV 2023)

      Volume: -

    • Related Report
      2022 Annual Research Report
    • Peer Reviewed
  • [Journal Article] Painting Style-Aware Manga Colorization Based On Generative Adversarial Networks2021

    • Author(s)
      Shimizu Yugo, Furuta Ryosuke, Ouyang Delong, Taniguchi Yukinobu, Hinami Ryota, Ishiwatari Shonosuke
    • Journal Title

      Proceedings of 2021 IEEE International Conference on Image Processing (ICIP 2021)

      Volume: - Pages: 1739-1743

    • DOI

      10.1109/icip42928.2021.9506254

    • Related Report
      2021 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Interactive Manga Colorization with Fast Flat Coloring2021

    • Author(s)
      Ouyang Delong, Furuta Ryosuke, Shimizu Yugo, Taniguchi Yukinobu, Hinami Ryota, Ishiwatari Shonosuke
    • Journal Title

      ACM SIGGRAPH Asia 2021 Posters

      Volume: - Pages: 1-2

    • DOI

      10.1145/3476124.3488628

    • Related Report
      2021 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Weakly-supervised Human-object Interaction Detection2021

    • Author(s)
      Masaki Sugimoto, Ryosuke Furuta, and Yukinobu Taniguchi
    • Journal Title

      Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

      Volume: 5 Pages: 293-300

    • DOI

      10.5220/0010196802930300

    • Related Report
      2020 Research-status Report
    • Peer Reviewed / Open Access
  • [Presentation] Detecting Mounting Behaviors of Dairy Cows by Pre-Training with Pseudo Images2023

    • Author(s)
      Y. Okuda, Y. Yamamoto, K. Nakamura and Y. Taniguchi
    • Organizer
      The 29th International Workshop on Frontiers of Computer Vision (IW-FCV 2023)
    • Related Report
      2022 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 手検出を用いた作業映像の教師なしアクション境界検出2023

    • Author(s)
      高木 利菜,山本 洋太,古田 諒佑,谷口 行信
    • Organizer
      画像電子学会第303回研究会
    • Related Report
      2022 Annual Research Report
  • [Presentation] 疑似マウンティング画像生成による乳牛の発情兆候検知2022

    • Author(s)
      奥田 悠太,山本 洋太,中村 和晃,谷口 行信
    • Organizer
      第50回画像電子学会年次大会
    • Related Report
      2022 Annual Research Report
  • [Presentation] ベタ塗り画像を利用した半自動漫画彩色手法2022

    • Author(s)
      清水友悟,古田諒佑,欧陽徳龍,谷口行信,日並遼太,石渡祥之佑
    • Organizer
      映像表現・芸術科学フォーラム 2022
    • Related Report
      2021 Research-status Report
  • [Presentation] Watershed法を用いたインタラクティブ漫画彩色システム2022

    • Author(s)
      欧陽徳龍,古田諒佑,清水友悟,谷口行信,日並遼太,石渡祥之佑
    • Organizer
      映像表現・芸術科学フォーラム 2022
    • Related Report
      2021 Research-status Report
  • [Presentation] 映像と文章の類似度学習による作業映像のシーン分割2021

    • Author(s)
      林純平,山本洋太,谷口行信
    • Organizer
      画像電子学会第297回研究会
    • Related Report
      2021 Research-status Report
  • [Presentation] Weakly-supervised Human-object Interaction Detection2021

    • Author(s)
      Masaki Sugimoto, Ryosuke Furuta, and Yukinobu Taniguchi
    • Organizer
      the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] 作業工程の言語情報を利用した作業映像のシーン分割2021

    • Author(s)
      林 純平,古田諒佑,谷口行信
    • Organizer
      電子情報通信学会2021年総合大会,学生ポスターセッション
    • Related Report
      2020 Research-status Report
  • [Presentation] 弱教師付き Human-Object Interaction 検出2020

    • Author(s)
      杉本将来,古田諒佑,谷口行信
    • Organizer
      映像情報メディア学会 創立70周年記念大会
    • Related Report
      2020 Research-status Report
  • [Presentation] 弱教師付きHuman Object Interaction検出2020

    • Author(s)
      杉本将来,古田諒佑,谷口行信
    • Organizer
      第23回画像の認識・理解シンポジウム(MIRU2020)
    • Related Report
      2020 Research-status Report
  • [Remarks] 東京理科大学 工学部 情報工学科 谷口研究室ホームページ

    • URL

      https://www.rs.tus.ac.jp/vml/

    • Related Report
      2022 Annual Research Report
  • [Remarks] 谷口研究室ホームページ

    • URL

      https://www.rs.tus.ac.jp/vml/

    • Related Report
      2021 Research-status Report

URL: 

Published: 2020-04-28   Modified: 2024-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi