• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Study on neural network for integrated processing of visual and linguistic information

Research Project

Project/Area Number 16K00338
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Research Field Soft computing
Research InstitutionKeio University

Principal Investigator

Hagiwara Masafumi  慶應義塾大学, 理工学部(矢上), 教授 (80198655)

Project Period (FY) 2016-04-01 – 2020-03-31
Project Status Completed (Fiscal Year 2019)
Budget Amount *help
¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000)
Fiscal Year 2018: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Fiscal Year 2017: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2016: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Keywords視覚情報処理 / 言語情報処理 / ニューラルネットワーク / 感性情報 / 対話システム / キャプション自動生成 / 転移学習 / 畳込みニューラルネットワーク / 分散表現 / 連想メモリ
Outline of Final Research Achievements

Regarding the development of integrated processing of visual information and linguistic information and the automatic acquisition of common sense from the input image, an automatic caption generation system is constructed. It can explain an image richly. The system utilizes not only the image features but also the noun information of the object when estimating the affective words of the object. It also has an affective word conversion mechanism to generate expressive words.
Concerning the development of neural networks that can generate natural language, we have constructed an automatic conversation system that can automatically generate conversational sentences considering the context. Furthermore, this was developed into an automatic consultation system focusing on empathy and advice.

Academic Significance and Societal Importance of the Research Achievements

従来より学術的には、画像のような視覚情報と言語情報は別々に扱われることが多かった。しかし実生活やTV、書籍、Webなど、ほとんど場合、これらは分離されることなく総合的に扱われている。
本研究の目的は、画像と言語を統合的に扱う方法を考案することにある。これは社会からの需要も高く、また学術的にも画像のようなパターン情報と言語のようなシンボル情報を統合処理するという意味で高い意義を有している。

Report

(5 results)
  • 2019 Annual Research Report   Final Research Report ( PDF )
  • 2018 Research-status Report
  • 2017 Research-status Report
  • 2016 Research-status Report
  • Research Products

    (16 results)

All 2019 2018 2017 2016

All Journal Article (9 results) (of which Peer Reviewed: 9 results,  Open Access: 7 results) Presentation (7 results) (of which Int'l Joint Research: 2 results,  Invited: 1 results)

  • [Journal Article] Dynamically Multilayered Neural Network2019

    • Author(s)
      佐々木駿也,萩原将文
    • Journal Title

      電子情報通信学会論文誌D 情報・システム

      Volume: J102-D Issue: 3 Pages: 226-234

    • DOI

      10.14923/transinfj.2018JDP7045

    • ISSN
      1880-4535, 1881-0225
    • Year and Date
      2019-03-01
    • Related Report
      2018 Research-status Report
    • Peer Reviewed
  • [Journal Article] Analog Value Associative Memory Using Restricted Boltzmann Machine2019

    • Author(s)
      Yuichiro Tsutsui and Masafumi Hagiwara
    • Journal Title

      Journal of Advanced Computational Intelligence and Intelligent Informatics

      Volume: 23 Issue: 1 Pages: 60-66

    • DOI

      10.20965/jaciii.2019.p0060

    • NAID

      130007563433

    • ISSN
      1343-0130, 1883-8014
    • Year and Date
      2019-01-20
    • Related Report
      2018 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Automatic Affective Image Captioning System using Emotion Estimation2019

    • Author(s)
      三由裕也, 萩原将文
    • Journal Title

      Transactions of Japan Society of Kansei Engineering

      Volume: 18 Issue: 2 Pages: 151-158

    • DOI

      10.5057/jjske.TJSKE-D-18-00071

    • NAID

      130007627245

    • ISSN
      1884-0833, 1884-5258
    • Related Report
      2019 Annual Research Report 2018 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Logo Generation with Generative Adversarial Networks Conditioned on Sentiment Terms2019

    • Author(s)
      佐川友里香,萩原将文
    • Journal Title

      Transactions of Japan Society of Kansei Engineering

      Volume: 18 Issue: 3 Pages: 215-222

    • DOI

      10.5057/jjske.TJSKE-D-18-00091

    • NAID

      130007638199

    • ISSN
      1884-0833, 1884-5258
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] A Proposal of Emotion Estimation Method for Words and Construction of Word-emotion Dictionary2019

    • Author(s)
      武内達哉,萩原将文
    • Journal Title

      Transactions of Japan Society of Kansei Engineering

      Volume: 18 Issue: 4 Pages: 273-278

    • DOI

      10.5057/jjske.TJSKE-D-18-00104

    • NAID

      130007697027

    • ISSN
      1884-0833, 1884-5258
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Automatic Consultation System by Latent Variable Hierarchical Recurrent Encoder-decoder using Transfer Learning2019

    • Author(s)
      和田翔,萩原将文
    • Journal Title

      Transactions of Japan Society of Kansei Engineering

      Volume: 18 Issue: 4 Pages: 307-314

    • DOI

      10.5057/jjske.TJSKE-D-19-00008

    • NAID

      130007697046

    • ISSN
      1884-0833, 1884-5258
    • Related Report
      2019 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] 対話システムにおけるランダム性を考慮した話題展開手法2018

    • Author(s)
      三上佳孝, 萩原将文
    • Journal Title

      日本感性工学研究論文集

      Volume: 17 Pages: 365-373

    • NAID

      130007381752

    • Related Report
      2018 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] 多様な応答のための分散表現ベース対話システム2018

    • Author(s)
      三上佳孝, 萩原将文
    • Journal Title

      日本感性工学研究論文集

      Volume: 17 Pages: 275-283

    • NAID

      130006728070

    • Related Report
      2018 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] 対話における文脈を考慮した分散表現生成ニューラルネットワーク2017

    • Author(s)
      三上佳孝,萩原将文
    • Journal Title

      日本知能情報ファジィ学会誌

      Volume: 29 Pages: 595-604

    • NAID

      130006893559

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Presentation] A Non-Task-Oriented Dialogue System Controlling the Utterance Length2018

    • Author(s)
      Kazuki Isoshima and Masafumi Hagiwara
    • Organizer
      International Conference on Soft Computing and Intelligent Systems & ISIS
    • Related Report
      2018 Research-status Report
    • Int'l Joint Research
  • [Presentation] 画像特徴と単語の分散表現を活用した感性語を付与した画像キャプションの自動生成2018

    • Author(s)
      瀬口太郎
    • Organizer
      電子情報通信学会ニューロコンピューティング研究会
    • Related Report
      2017 Research-status Report
  • [Presentation] 段階的に多層化するニューラルネットワーク2018

    • Author(s)
      佐々木駿也
    • Organizer
      電子情報通信学会ニューロコンピューティング研究会
    • Related Report
      2017 Research-status Report
  • [Presentation] 分散表現を用いたRBM連想メモリによる知識処理2017

    • Author(s)
      椎名健之
    • Organizer
      電子情報通信学会ニューロコンピューティング研究会
    • Place of Presentation
      機械振興会館(東京都・港区)
    • Year and Date
      2017-03-13
    • Related Report
      2016 Research-status Report
  • [Presentation] Continuous-Value Associative Memory Using Restricted Boltzmann Machine2017

    • Author(s)
      Masafumi Hagiwara
    • Organizer
      International Conference on Soft Computing and Intelligent Systems (ISIS 2017)
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research
  • [Presentation] 心を考慮した人工知能へ ~知情意の考慮~2017

    • Author(s)
      萩原将文
    • Organizer
      第50回日本人間工学会中国・四国支部大会
    • Related Report
      2017 Research-status Report
    • Invited
  • [Presentation] 連想層と抑制層を導入した言語処理ニューラルネットワーク2016

    • Author(s)
      山本眞大
    • Organizer
      第32回ファジィシステムシンポジウム
    • Place of Presentation
      佐賀大学(佐賀県・佐賀市)
    • Year and Date
      2016-09-01
    • Related Report
      2016 Research-status Report

URL: 

Published: 2016-04-21   Modified: 2021-02-19  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi