learning of multimodal representation and its application

• Project/Area Number: 26330249
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Intelligent informatics
• Research Institution: Shinshu University
• Principal Investigator: MARUYAMA Minoru 信州大学, 学術研究院工学系, 教授 (80283232)
• Project Period (FY): 2014-04-01 – 2017-03-31
• Project Status: Completed (Fiscal Year 2016)
• Budget Amount: ¥4,550,000 (Direct Cost: ¥3,500,000、Indirect Cost: ¥1,050,000); Fiscal Year 2016: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000); Fiscal Year 2015: ¥2,080,000 (Direct Cost: ¥1,600,000、Indirect Cost: ¥480,000); Fiscal Year 2014: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
• Keywords: 画像認識 / 機械学習 / 深層学習 / 画像検索 / ニューラルネットワーク / 画像識別 / ネットワーク圧縮 / 属性認識 / 構造学習 / 確率的トピックモデル / 文書認識 / 多重音解析 / 画像想起性

Outline of Final Research Achievements

Research on multimodal learning and its application to image search has been carried out. We have studied the DBM to jointly represent the image and corresponding text. We showed feature vector obtained from the joint layer could give rise to better classification results than CNN-based features. We also studied image query method based on multimodal representation which is enabled by using visual-semantic embedding model based on CNN and LSTM. It allows us to perform analogical reasoning over images by specifying properties to be added and subtracted by words. We introduced a novel similarity measure based on the difference between additive and subtractive query. Our methods strongly depend upon the CNNs for image processing. To reduce the computational cost of the image processing by the CNN, we examined a method for compressing the given CNN. We proposed the compression method based on the block-wise distillation and examined its effectiveness.

Research Products

• [Presentation] Inetractive image search system based on multimodal analogy (2017)
  • Author(s): Kosuke Ota, Keiichiro Shirai, Hidetoshi Miyao, Minoru Maruyama
  • Organizer: 19th International Conference on Human-Computer Interaction
  • Place of Presentation: Vancouver Convention Centre, Vancouver, Canada
  • Year and Date: 2017-07-12
  • Int'l Joint Research
• [Presentation] Multimodal representation learning by utilizing deep convolutional features and preprocessed texts (2016)
  • Author(s): Kosuke Ota, Keiichiro Shirai, Hidetoshi Miyao, Minoru Maruyama
  • Organizer: IEEE Shin-etsu session
  • Place of Presentation: 長岡技術科学大
  • Year and Date: 2016-10-08
• [Presentation] 既存深層構造に対する圧縮方法の検討 (2016)
  • Author(s): 横島府, 白井啓一郎, 宮尾秀俊, 丸山稔
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 長岡技術科学大
  • Year and Date: 2016-10-08
• [Presentation] 既存深層構造を用いた新規カテゴリ学習方式に関する検討 (2015)
  • Author(s): 横島府, 白井啓一郎, 宮尾秀俊, 丸山稔
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 新潟工科大学
  • Year and Date: 2015-10-03
• [Presentation] 画像の属性識別のためのConvolutional Neural Networksの学習 (2015)
  • Author(s): 宇都宮気伸, 白井啓一郎, 宮尾秀俊, 丸山稔
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 新潟工科大学
  • Year and Date: 2015-10-03
• [Presentation] 画像の想起性識別のための特徴量選択 (2014)
  • Author(s): 宇都宮気伸, 丸山稔, 宮尾秀俊
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 信州大学工学部（長野県・長野市）
  • Year and Date: 2014-10-04
• [Presentation] 識別関数の学習に基づくピアノ音楽の自動採譜 (2014)
  • Author(s): 福島崚平, 宮尾秀俊, 丸山稔
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 信州大学工学部（長野県・長野市）
  • Year and Date: 2014-10-04
• [Presentation] 確率的トピックモデルを用いた論文データの経年変化解析 (2014)
  • Author(s): 桜井翔, 宮尾秀俊, 丸山稔
  • Organizer: 電子情報通信学会信越支部大会
  • Place of Presentation: 信州大学工学部（長野県・長野市）
  • Year and Date: 2014-10-04