Study on neural network for integrated processing of visual and linguistic information
Project/Area Number |
16K00338
|
Research Category |
Grant-in-Aid for Scientific Research (C)
|
Allocation Type | Multi-year Fund |
Section | 一般 |
Research Field |
Soft computing
|
Research Institution | Keio University |
Principal Investigator |
|
Project Period (FY) |
2016-04-01 – 2020-03-31
|
Project Status |
Completed (Fiscal Year 2019)
|
Budget Amount *help |
¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000)
Fiscal Year 2018: ¥1,690,000 (Direct Cost: ¥1,300,000、Indirect Cost: ¥390,000)
Fiscal Year 2017: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Fiscal Year 2016: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
|
Keywords | 視覚情報処理 / 言語情報処理 / ニューラルネットワーク / 感性情報 / 対話システム / キャプション自動生成 / 転移学習 / 畳込みニューラルネットワーク / 分散表現 / 連想メモリ |
Outline of Final Research Achievements |
Regarding the development of integrated processing of visual information and linguistic information and the automatic acquisition of common sense from the input image, an automatic caption generation system is constructed. It can explain an image richly. The system utilizes not only the image features but also the noun information of the object when estimating the affective words of the object. It also has an affective word conversion mechanism to generate expressive words. Concerning the development of neural networks that can generate natural language, we have constructed an automatic conversation system that can automatically generate conversational sentences considering the context. Furthermore, this was developed into an automatic consultation system focusing on empathy and advice.
|
Academic Significance and Societal Importance of the Research Achievements |
従来より学術的には、画像のような視覚情報と言語情報は別々に扱われることが多かった。しかし実生活やTV、書籍、Webなど、ほとんど場合、これらは分離されることなく総合的に扱われている。 本研究の目的は、画像と言語を統合的に扱う方法を考案することにある。これは社会からの需要も高く、また学術的にも画像のようなパターン情報と言語のようなシンボル情報を統合処理するという意味で高い意義を有している。
|
Report
(5 results)
Research Products
(16 results)