• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

A study on shared representation learning considering the uncertainty of each modality

Research Project

Project/Area Number 19K21527
Project/Area Number (Other) 18H06458 (2018)
Research Category

Grant-in-Aid for Research Activity Start-up

Allocation TypeMulti-year Fund (2019)
Single-year Grants (2018)
Review Section 1001:Information science, computer engineering, and related fields
Research InstitutionThe University of Tokyo

Principal Investigator

Suzuki Masahiro  東京大学, 大学院工学系研究科(工学部), 特任研究員 (30823885)

Project Period (FY) 2018-08-24 – 2020-03-31
Project Status Completed (Fiscal Year 2019)
Budget Amount *help
¥2,990,000 (Direct Cost: ¥2,300,000、Indirect Cost: ¥690,000)
Fiscal Year 2019: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2018: ¥1,560,000 (Direct Cost: ¥1,200,000、Indirect Cost: ¥360,000)
Keywords深層学習 / 共有表現学習 / マルチモーダル学習 / 深層生成モデル
Outline of Research at the Start

深層学習の飛躍的発展によって、画像や音声、文書といった異なる種類の情報(モダリティ)を統合して学習するマルチモーダル学習の研究が数多く行われるようになっている。マルチモーダル学習は、異なるモダリティを統合した共有表現の学習が重要となる。しかし異なるモダリティ間で1対1の決定論的な対応関係を結べないような場合(例えば1つのタグに対応する画像は無数にある)、従来の決定論的な学習方法では適切な共有表現を獲得できない。本研究では、申請者らが開発した深層生成モデルによる手法に基づき、各モダリティの不確実性を考慮することで、複数のモダリティ情報を統合した共有表現を適切に学習する手法を確立する。

Outline of Final Research Achievements

In this research, we addressed how to integrate several different types of information (i.e., different modalities), such as images, documents, and sounds. Previous studies did not take into account the differences in uncertainty across modalities and therefore integrated them deterministically. In this study, we proposed the probabilistic integration of different modalities based on a framework called deep generative models. We then showed that this approach is effective in multiple multimodal learning problem settings. In addition, we developed a new library to simplify the implementation of complex deep generative models containing multimodal information relationships.

Academic Significance and Societal Importance of the Research Achievements

本研究で提案する異なるモダリティの統合の枠組みは,今回扱ったデータや問題設定によらず,様々な領域に応用できると考えている.それは,この統合方法では深層生成モデルを用いてるため,モダリティの不確実性の違いのみに着目しており,モダリティの入力空間の次元数には依存しないからである.また,今回開発した深層生成モデルライブラリは,マルチモーダル学習のモデルに限らず,様々な深層生成モデルの実装に利用することができる.

Report

(3 results)
  • 2019 Annual Research Report   Final Research Report ( PDF )
  • 2018 Annual Research Report
  • Research Products

    (11 results)

All 2020 2019 2018

All Journal Article (3 results) (of which Int'l Joint Research: 1 results,  Peer Reviewed: 3 results,  Open Access: 1 results) Presentation (8 results) (of which Int'l Joint Research: 4 results,  Invited: 1 results)

  • [Journal Article] Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models2020

    • Author(s)
      Tadahiro Taniguchi, Tomoaki Nakamura, Masahiro Suzuki, Ryo Kuniyasu, Kaede Hayashi, Akira Taniguchi, Takato Horii, Takayuki Nagai
    • Journal Title

      New Generation Computing

      Volume: 38 Issue: 1 Pages: 23-48

    • DOI

      10.1007/s00354-019-00084-w

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] 服の領域を考慮した写真上の人物の自動着せ替えに関する研究2019

    • Author(s)
      久保静真,岩澤有祐,鈴木雅大,松尾豊
    • Journal Title

      情報処理学会論文誌

      Volume: 60 Pages: 870-879

    • NAID

      170000150210

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Journal Article] 深層生成モデルを用いた半教師ありマルチモーダル学習2018

    • Author(s)
      鈴木雅大,松尾豊
    • Journal Title

      情報処理学会論文誌

      Volume: 59 Pages: 2261-2278

    • NAID

      170000149944

    • Related Report
      2018 Annual Research Report
    • Peer Reviewed
  • [Presentation] Pixyz: a framework for developing complex deep generative models2019

    • Author(s)
      Masahiro Suzuki
    • Organizer
      Workshop on Deep Probabilistic Generative Models for Cognitive Architecture in Robotics (IROS2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] 深層生成モデルと世界モデル2019

    • Author(s)
      鈴木雅大
    • Organizer
      第4回統計・機械学習若手シンポジウム
    • Related Report
      2019 Annual Research Report
  • [Presentation] UVTON: UV Mapping to Consider the 3D Structure of a Human in Image-Based Virtual Try-On Network2019

    • Author(s)
      Shizuma Kubo, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo
    • Organizer
      Workshop on Computer Vision for Fashion, Art and Design, The IEEE International Conference on Computer Vision (ICCV 2019)
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Dual Space Learning with Variational Autoencoders2019

    • Author(s)
      Hirono Okamoto, Masahiro Suzuki, Itto Higuchi, Shohei Ohsawa, Yutaka Matsuo
    • Organizer
      Workshop on Deep Generative Models for Highly Structured Data, International Conference on Learning Representation
    • Related Report
      2019 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Dual Space Learning with Variational Autoencoders2019

    • Author(s)
      Hirono Okamoto, Masahiro Suzuki, Itto Higuchi, Shohei Ohsawa, Yutaka Matsuo
    • Organizer
      ICLR workshop
    • Related Report
      2018 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Pixyz:複雑な深層生成モデル開発のためのフレームワーク2019

    • Author(s)
      鈴木 雅大, 金子 貴輝, 谷口 尚平, 松嶋 達也, 松尾 豊
    • Organizer
      2019年度人工知能学会全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 身体の3次元構造を考慮したニューラル仮想試着2019

    • Author(s)
      久保 静真, 岩澤 有祐, 鈴木 雅大, 松尾 豊
    • Organizer
      2019年度人工知能学会全国大会
    • Related Report
      2018 Annual Research Report
  • [Presentation] 半教師ありマルチモーダル深層生成モデルにおける共有表現の有効性と単一モダリティ入力への拡張2018

    • Author(s)
      鈴木雅大,松尾豊
    • Organizer
      2018年度人工知能学会全国大会
    • Related Report
      2018 Annual Research Report

URL: 

Published: 2018-08-27   Modified: 2024-03-26  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi