• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Formal Semantic Representations to Link Language and Vision

Research Project

Project/Area Number 18H03268
Research Category

Grant-in-Aid for Scientific Research (B)

Allocation TypeSingle-year Grants
Section一般
Review Section Basic Section 61010:Perceptual information processing-related
Research InstitutionThe University of Tokyo

Principal Investigator

Miyao Yusuke  東京大学, 大学院情報理工学系研究科, 教授 (00343096)

Project Period (FY) 2018-04-01 – 2021-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥17,160,000 (Direct Cost: ¥13,200,000、Indirect Cost: ¥3,960,000)
Fiscal Year 2020: ¥5,590,000 (Direct Cost: ¥4,300,000、Indirect Cost: ¥1,290,000)
Fiscal Year 2019: ¥6,110,000 (Direct Cost: ¥4,700,000、Indirect Cost: ¥1,410,000)
Fiscal Year 2018: ¥5,460,000 (Direct Cost: ¥4,200,000、Indirect Cost: ¥1,260,000)
Keywords意味表現 / 自然言語処理 / 画像処理
Outline of Final Research Achievements

This research explored semantic representations for images with the aim of applying semantic analysis technologies of natural languages to visual information. Specifically, we developed a method for linking entities in an input image into database IDs and a technique for compositionally constructing semantic representations of images. In addition, we designed a new task of generating a caption given an image and a fragment of a semantic representation as input and showed the effectiveness of using semantic representations for images.

Academic Significance and Societal Importance of the Research Achievements

画像と言語をつなぐ技術は近年数多く研究されているが、そのほとんどは画像と言語を入出力として深層学習モデルを学習する手法である。この手法は大規模な学習データがあれば多くのタスクで高い精度を達成するが、学習データがない場合や、外部知識や推論を必要とする高度なタスクに適用することは難しい。提案手法のように画像に対して意味表現を得ることができれば、意味表現を利用した自然言語処理技術を応用する道が開け、さまざまな技術に発展することが期待できる。

Report

(4 results)
  • 2021 Final Research Report ( PDF )
  • 2020 Annual Research Report
  • 2019 Annual Research Report
  • 2018 Annual Research Report

Research Products

(2 results)

All 2021 2019

All Journal Article (2 results) (of which Peer Reviewed: 1 results,  Open Access: 1 results)

  • [Journal Article] Leveraging Partial Dependency Trees to Control Image Captions2021

    • Author(s)
      Zhong Wenjie、Miyao Yusuke
    • Journal Title

      Proceedings of the Second Workshop on Advances in Language and Vision Research

      Volume: 1 Pages: 16-21

    • DOI

      10.18653/v1/2021.alvr-1.3

    • Related Report
      2020 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] 多様なデータと自然言語をつなぐ基盤技術2019

    • Author(s)
      宮尾 祐介
    • Journal Title

      学会誌「人工知能」特集「人間と相互理解できる次世代人工知能技術」

      Volume: 34(6) Pages: 811-816

    • NAID

      130007917664

    • Related Report
      2019 Annual Research Report

URL: 

Published: 2018-04-23   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi