Flora and fauna identification system based on multi-modal data observed

• Project/Area Number: 15K12027
• Research Category: Grant-in-Aid for Challenging Exploratory Research
• Allocation Type: Multi-year Fund
• Research Field: Multimedia database
• Research Institution: Toyohashi University of Technology
• Principal Investigator: Aono Masaki 豊橋技術科学大学, 工学(系)研究科(研究院), 教授 (00372540)
• Project Period (FY): 2015-04-01 – 2018-03-31
• Project Status: Completed (Fiscal Year 2017)
• Budget Amount: ¥3,510,000 (Direct Cost: ¥2,700,000、Indirect Cost: ¥810,000); Fiscal Year 2017: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000); Fiscal Year 2016: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000); Fiscal Year 2015: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
• Keywords: 植物鑑定 / 深層学習 / 画像特徴量 / 端数型プーリング / ノイズ有訓練データ / 双線形フィルタ / データ圧縮

Outline of Final Research Achievements

From the plant photo images posted on the Internet, We have participated in the international contest called PlantCLEF from the second year of the research period, resulting in No.1 world record in 2016 and the year 2017, "Noisy training data" truck, which was the second place as a whole. Technically, we have succeeded in improving the accuracy by mixing textual features from meta data with the image features of CNN and MLP layers of the deep learning as well as the traditional features such as Fisher vector. Furthermore, we have developed a fractional pooling method to enable convolution with arbitrary rectangular images.

Research Products

• [Journal Article] Super-vector coding features extracted from both depth buffer and view-normal-angle images for part-based 3D shape retrieval (2017)
  • Author(s): Ｓhoki Tashiro, Atsushi Tatsuma, and Masaki Aono
  • Journal Title: Multimedia Tools and Applications Volume: 76 Issue: 21 Pages: 22059-22076
  • DOI: 10.1007/s11042-017-4801-z
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Volumetric Spiral Scan Orderによる三次元考古遺物モデルの類似検索 (2017)
  • Author(s): 濵田和真,立間淳司, 青野雅樹
  • Journal Title: 情報処理学会論文誌 Volume: 10 Pages: 1-7
  • NAID: 170000148699
  • Peer Reviewed
• [Journal Article] Microblog Retrieval Using Ensemble of Feature Sets through Supervised Feature Selection (2017)
  • Author(s): Abu Nowshed Chy, Md Zia Ullah and Masaki Aono
  • Journal Title: IEICE Transactions on Information and Systems Volume: E100.D Issue: 4 Pages: 793-806
  • DOI: 10.1587/transinf.2016DAP0032
  • NAID: 130005529887
  • ISSN: 0916-8532, 1745-1361
  • Peer Reviewed / Open Access / Int'l Joint Research / Acknowledgement Compliant
• [Journal Article] Bi-linearly weighted fractional max pooling (2017)
  • Author(s): Siang Thye Hang and Masaki Aono
  • Journal Title: Multimedia Tools and Applications Volume: 76 Issue: 21 Pages: 22095-22117
  • DOI: 10.1007/s11042-017-4840-5
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Shape Retrieval of Non-rigid 3D Human Models (2016)
  • Author(s): D.Pickup, X.Sun, P.L.Rosin, R.R.Martin, Z.Cheng, Z.Lian, Masaki Aono
  • Journal Title: International Journal of Computer Vision Volume: 120 Issue: 2 Pages: 169-193
  • DOI: 10.1007/s11263-016-0903-8
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] A Bipartite Graph-Based Ranking Approach to Query Subtopics Diversification Focused on Word Embedding Features (2016)
  • Author(s): Md Zia Ullah and Masaki Aono
  • Journal Title: IEICE Transactions on Information and Systems Volume: E99.D Issue: 12 Pages: 3090-3100
  • DOI: 10.1587/transinf.2016EDP7190
  • NAID: 130005170962
  • ISSN: 0916-8532, 1745-1361
  • Peer Reviewed / Open Access / Int'l Joint Research / Acknowledgement Compliant
• [Journal Article] Food Image Recognition Using Covariance of Convolutional Layer Feature Maps (2016)
  • Author(s): Atsushi Tatsuma and Masaki Aono
  • Journal Title: IEICE Transactions on Information and Systems Volume: E99.D Issue: 6 Pages: 1711-1715
  • DOI: 10.1587/transinf.2015EDL8212
  • NAID: 130005154713
  • ISSN: 0916-8532, 1745-1361
  • Peer Reviewed / Open Access
• [Journal Article] Retrieval and classification methods for textured 3D models: A comparative study (2016)
  • Author(s): S. Biasotti, A. Cerri, Masaki Aono, A. Ben Hamza, V. Garro, A. Giachetti, D. Giorgi, A. Godil, C. Li , Chika Sanada, M. Spagnuolo, Atsushi Tatsuma, S. Velasco-Forero
  • Journal Title: Visual Computer Volume: 32 Issue: 2 Pages: 217-241
  • DOI: 10.1007/s00371-015-1146-3
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Estimating a Ranked List of Human Genetic Diseases by Associating Phenotype-Gene with Gene-Disease Bipartite Graphs (2015)
  • Author(s): Md Zia Ullah, Masaki Aono, Md Hanif Seddiqui
  • Journal Title: ACM Transactions on Intelligent Systems and Technology Volume: 6 Issue: 4 Pages: 1-21
  • DOI: 10.1145/2700487
  • Peer Reviewed / Int'l Joint Research
• [Presentation] 画像へのマルチタグとそれらの重要度の自動付与 (2018)
  • Author(s): 杉山裕哉, 青野雅樹
  • Organizer: 第10回データ工学と情報マネジメントに関するフォーラム(DEIM2018)
• [Presentation] 局所特徴量を用いた3D CNNによる3 次元モデルの部分検索 (2018)
  • Author(s): 岩淵渉, 青野雅樹
  • Organizer: 第10回データ工学と情報マネジメントに関するフォーラム(DEIM2018)
• [Presentation] ボクセル群の奥行き密度反映した画像による屋内3Dシーン分類 (2018)
  • Author(s): 濵田和真, 青野雅樹
  • Organizer: パターン認識・メディア理解研究会（PRMU2017-204 (2018-3)）
• [Presentation] エッジと輝度反転に着目した特徴量とCNN特徴量を用いたマンガ画像の著者推定 (2018)
  • Author(s): 竹ノ内達哉, 青野雅樹
  • Organizer: 電子情報通信学会総合大会
• [Presentation] Sliced Voxel Representations with LSTM and CNN for 3D Shape Recognition (2017)
  • Author(s): Ryo Miyagi and Masaki Aono
  • Organizer: Asia Pacific Signal and Information Processing Association (APSIPA2017)
  • Int'l Joint Research
• [Presentation] Residual Network with Delayed Max Pooling for Very Large Scale Plant Identification (2017)
  • Author(s): Siang Thye Hang and Masaki Aono
  • Organizer: LifeCLEF 2017 Workshop in Conference and Labs of the Evaluation Forum (CLEF2017)
  • Int'l Joint Research
• [Presentation] An Attention-based Regression Model for Grounding Textual Phrases in Images (2017)
  • Author(s): Ko Endo, Masaki Aono, Eric Nichols, Kotarou Funakoshi
  • Organizer: International Joint Conference on Artificial Intelligence (IJCAI2017)
  • Int'l Joint Research
• [Presentation] Predicting the Usefulness of Cosmetic Reviews (2017)
  • Author(s): Yuri Takashima and Masaki Aono
  • Organizer: International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA2017)
  • Int'l Joint Research
• [Presentation] Tweet Polarity Classification Focused on Positive and Negative Term Frequency Ratio (2017)
  • Author(s): Shinnosuke Himeno and Masaki Aono
  • Organizer: International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA2017)
  • Int'l Joint Research
• [Presentation] Point-Cloud Shape Retrieval of Non-Rigid Toys (2017)
  • Author(s): Limberger, F. A. and Aono, M. et al.
  • Organizer: Eurographics Workshop on 3D Object Retrieval
  • Int'l Joint Research
• [Presentation] RGB-D to CAD Retrieval with ObjectNN Dataset (2017)
  • Author(s): Binh-Son Hua, Asako Kanezaki, Bo Li, Yijuan Lu, Shoki Tashiro, Masaki Aono, et al.
  • Organizer: Eurographics Workshop on 3D Object Retrieval
  • Int'l Joint Research
• [Presentation] Large-Scale 3D Shape Retrieval from ShapeNet Core55 (2017)
  • Author(s): Manolis Savva, Asako Kanezaki, Takahiko Furuya, Ryutarou Ohbuchi, Masaki Aono, Atsushi Tatsuma, S. Thermos, A. Axenopoulos, G. Th. Papadopoulos, P. Daras, Xiao Deng, Zhouhui Lian, Bo Li, st al.
  • Organizer: Eurographics Workshop on 3D Object Retrieval
  • Int'l Joint Research
• [Presentation] Deep Spatio-Temporal Transformationを用いた逐次人物動作検出 (2017)
  • Author(s): 髙垣幸秀, 青野雅樹
  • Organizer: パターン認識・メディア理解研究会（PRMU 2017-68)
• [Presentation] 異方性を考慮した画像の符号化に基づく時系列データ分類 (2017)
  • Author(s): 岩室伸哉, 青野雅樹
  • Organizer: パターン認識・メディア理解研究会（PRMU 2017-56)
• [Presentation] 単語と文の分散表現素性に着目したニュースストリームに対する文単位の新規性判定 (2017)
  • Author(s): 田村壮慶, 青野雅樹, 立間淳司
  • Organizer: 第16回情報科学技術フォーラム(FIT2017)
• [Presentation] 単語分散表現と文法的な表現に着目したカスタマーレビューの観点ごとの評価値推定 (2017)
  • Author(s): 大塚 達也, 青野雅樹
  • Organizer: DEIM2017 (第9回データ工学と情報マネジメントに関するフォーラム）
  • Place of Presentation: 高山グリーンホテル, 岐阜県
• [Presentation] LSTMとCNNを用いたボクセル表現に基づく三次元形状類似検索手法の提案 (2017)
  • Author(s): 宮城諒, 青野雅樹
  • Organizer: パターン認識・メディア理解(PRMU)研究会
  • Place of Presentation: 名城大学, 愛知県
• [Presentation] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task (2016)
  • Author(s): Siang Thye Hang, Atsushi Tatsuma, and Masaki Aono
  • Organizer: CLEF2016
  • Place of Presentation: University of Évora, Portugal
  • Year and Date: 2016-09-05
  • Int'l Joint Research
• [Presentation] 画像内領域の顕著性を考慮した画像の説明文生成 (2016)
  • Author(s): 吉井和輝, Eric Nichols, 船越孝太郎, 中野幹生, 青野雅樹
  • Organizer: 電子情報通信学会パターン認識・メディア理解研究会
  • Place of Presentation: 産業総合研究所(臨海副都心センター)
  • Year and Date: 2016-03-24
• [Presentation] 時間軸を考慮した特徴量の提案とそれを用いた動画分類 (2016)
  • Author(s): 髙垣幸秀, 青野雅樹
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 九州大学伊都キャンパス
  • Year and Date: 2016-03-16
• [Presentation] 重要単語からなる料理テキストと料理画像のマルチモーダル特徴量を用いた料理名推定 (2016)
  • Author(s): 小田将規, 青野雅樹
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 九州大学伊都キャンパス
  • Year and Date: 2016-03-15
• [Presentation] Authorship Identification focused on Sentence and Document Based Features (2016)
  • Author(s): Shofi Nur Fathiya and Masaki Aono
  • Organizer: 電子情報通信学会総合大会
  • Place of Presentation: 九州大学伊都キャンパス
  • Year and Date: 2016-03-15
• [Presentation] 色付き三次元物体の類似検索 (2016)
  • Author(s): 真田知佳，立間淳司，青野雅樹
  • Organizer: 情報処理学会第78回全国大会
  • Place of Presentation: 慶応大学矢上キャンパス
  • Year and Date: 2016-03-10
• [Presentation] 顕著性マップを用いた画像の説明文自動生成 (2016)
  • Author(s): 吉井和輝, エリック・ニコルズ, 船越孝太郎, 中野幹生, 青野雅樹
  • Organizer: 人工知能学会全国大会
  • Place of Presentation: 北九州国際会議センター
• [Presentation] テキストクエリで指定した画像中の領域を特定する一手法 (2016)
  • Author(s): 遠藤 昂, 船越孝太郎, エリック・ニコルズ, 青野雅樹
  • Organizer: 電子情報通信学会 画像工学研究会
  • Place of Presentation: 福岡大学
• [Presentation] Open World Plant Image Identification Based on Convolutional Neural Network (2016)
  • Author(s): Siang Thye Hang and Masaki Aono
  • Organizer: Asia Pacific Signal and Information Processing Association (APSIPA2016)
  • Place of Presentation: Jeju, Korea
  • Int'l Joint Research
• [Presentation] Query Subtopic Mining Exploiting Word Embedding for Search Result Diversification (2016)
  • Author(s): Md. Zia Ullah, Md. Shajalal, Abu Nowshed Chy, and Masaki Aono
  • Organizer: Asia Information Retrieval Societies Conference (AIRS2016)
  • Place of Presentation: Beijing, China
  • Int'l Joint Research
• [Presentation] 3D Shape Retrieval from a Photo Using Intrinsic Image (2015)
  • Author(s): Shoki Tashiro and Masaki Aono
  • Organizer: Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA2015)
  • Place of Presentation: Hong Kong
  • Year and Date: 2015-12-19
  • Int'l Joint Research
• [Presentation] 文法的な表現を手がかりとした宿泊施設レビュー文の意見分類 (2015)
  • Author(s): 大塚達也, 立間淳司, 青野雅樹
  • Organizer: 第14回情報科学技術フォーラム (FIT2015)
  • Place of Presentation: 愛媛大学
  • Year and Date: 2015-09-17
• [Presentation] 料理レシピサイトから抽出される特徴に基づいた調理時間予測 (2015)
  • Author(s): 阿部卓也, 立間淳司, 青野雅樹
  • Organizer: 第14回情報科学技術フォーラム (FIT2015)
  • Place of Presentation: 愛媛大学
  • Year and Date: 2015-09-16
• [Presentation] Combining Temporal and Content Aware Features for Microblog Retrieval (2015)
  • Author(s): Abu Nowshed Chy, Md Zia Ullah and Masaki Aono
  • Organizer: The 2015 International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA 2015)
  • Place of Presentation: Bang Saen Beach, Chonburi, Thailand
  • Year and Date: 2015-08-20
  • Int'l Joint Research
• [Presentation] New Composite Shape and Texture Descriptors for 3D Model Retrieval (2015)
  • Author(s): Hero Yudo Martono, Masaki Aono
  • Organizer: 4th International Conference on Computing and Computer Vision (ICCCV2015)
  • Place of Presentation: Hong Kong
  • Year and Date: 2015-06-23
• [Presentation] Time and Context Aware Re-ranker for Microblog Retrieval (2015)
  • Author(s): Chy Abu Nowshed, Ullah Md Zia, 青野 雅樹
  • Organizer: 人工知能学会全国大会
  • Place of Presentation: はこだて未来大学
  • Year and Date: 2015-05-31
• [Presentation] 日本語単語ベクトルの構築とその評価 (2015)
  • Author(s): 吉井和輝，ニコルズエリック，中野 幹生, 青野 雅樹
  • Organizer: 情報処理学会第221回自然言語処理研究会
  • Place of Presentation: 東北大学
  • Year and Date: 2015-05-25
• [Patent(Industrial Property Rights)] 三次元画像分類装置および三次元画像分類方法 (2018)
  • Inventor(s): 濵田和真、青野雅樹
  • Industrial Property Rights Holder: 濵田和真、青野雅樹
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2018-046791
  • Filing Date: 2018
• [Patent(Industrial Property Rights)] 検出装置および検出方法 (2017)
  • Inventor(s): 髙垣幸秀、青野雅樹
  • Industrial Property Rights Holder: 髙垣幸秀、青野雅樹
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2017-197694
  • Filing Date: 2017
• [Patent(Industrial Property Rights)] 三次元モデル検索方法及び三次元モデル検索システム (2017)
  • Inventor(s): 小林祐輝、青野雅樹
  • Industrial Property Rights Holder: 小林祐輝、青野雅樹
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2017-029425
  • Filing Date: 2017-02-20
• [Patent(Industrial Property Rights)] 画像特徴量及びそれを用いる三次元形状検索システム (2017)
  • Inventor(s): 立間淳司、小林将也、青野雅樹
  • Industrial Property Rights Holder: 立間淳司、小林将也、青野雅樹
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2017-042156
  • Filing Date: 2017-03-06
• [Patent(Industrial Property Rights)] 画像認識装置、画像認識方法、及び画像認識プログラム (2016)
  • Inventor(s): 立間淳司、青野雅樹
  • Industrial Property Rights Holder: 立間淳司、青野雅樹
  • Industrial Property Rights Type: 特許
  • Industrial Property Number: 2016-008273
  • Filing Date: 2016-01-19