Development of New Generation Artificial Intelligence Based on Theoretical Analyses of Deep Learning

• Project/Area Number: 18K19821
• Research Category: Grant-in-Aid for Challenging Research (Exploratory)
• Allocation Type: Multi-year Fund
• Review Section: Medium-sized Section 61:Human informatics and related fields
• Research Institution: Nara Institute of Science and Technology
• Principal Investigator: Ikeda Kazushi 奈良先端科学技術大学院大学, 先端科学技術研究科, 教授 (10262552)
• Project Period (FY): 2018-06-29 – 2022-03-31
• Project Status: Completed (Fiscal Year 2021)
• Budget Amount: ¥6,240,000 (Direct Cost: ¥4,800,000、Indirect Cost: ¥1,440,000); Fiscal Year 2020: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000); Fiscal Year 2019: ¥2,470,000 (Direct Cost: ¥1,900,000、Indirect Cost: ¥570,000); Fiscal Year 2018: ¥2,470,000 (Direct Cost: ¥1,900,000、Indirect Cost: ¥570,000)
• Keywords: 深層学習 / ResNet / 汎化ギャップ / 学習曲線 / truncated data / Barron空間 / トラッキング / グラフ信号処理 / スキップコネクション / ドロップアウト / ストキャスティックデプス / 機械学習 / 人工知能 / 統計的学習理論 / 学習理論 / 統計的漸近論 / 情報幾何学

Outline of Final Research Achievements

The purpose of this project is to analyze the effects of heuristic techniques for deep learning from the theoretical viewpoint since the theoretical results give insight to design a better/more efficient neural network/learning algorithm. Our analysis on a ResNet shows that shortcuts in a ResNet decrease an upper bound of the generalization cap and that a ResNet with two-layer skips has a lower upper-bound than one with one-layer skips. Some other heuristics are analyzed and they are also useful for the design of deep learning.

Academic Significance and Societal Importance of the Research Achievements

深層学習がこれほど注目されているにもかかわらず、その理論的背景は明らかではなかった。その原因の一つは、深層学習は構造的には1980 年代のネオコグニトロンと同等であり、その学習方法にプレトレーニングやドロップアウト，あるいは確率降下法などのヒューリスティックな工夫を組み合わせたもので、どの要素がどのような効果を持つのかがわかりにくかったためである。日本神経回路学会などでもBeyond Deep Learningが議論されるなど、次世代脳型人工知能の開発にはまず深層学習の仕組みを明らかにする必要があり、本研究によりいくつかの要素技術の本質的な役割が解明された。

Research Products

• [Int'l Joint Research] Ateneo de Manila University/De La Salle University/Augmented Intelligence Pros(フィリピン)
• [Journal Article] Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities (2022)
  • Author(s): Takeshi D. Itoh, Takatomi Kubo, Kazushi Ikeda
  • Journal Title: Neural Networks Volume: 145 Pages: 356-373
  • DOI: 10.1016/j.neunet.2021.11.001
  • Peer Reviewed / Open Access
• [Journal Article] The Relative Isolation Probability of a Vertex in a Multiple-Source Edge-Weighted Graph (2022)
  • Author(s): Renzo Roel P. Tan, Kyle Stephen S. See, Kazushi Ikeda, Jun Kawahara, Richard M. de Jesus, Lessandro Estelito O. Garciano, Agnes D. Garciano
  • Journal Title: Engineering Letters Volume: 10
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Concerning a Decision-Diagram-Based Solution to the Generalized Directed Rural Postman Problem (2020)
  • Author(s): Renzo Roel P. Tan, Jun Kawahara, Kazushi Ikeda, Agnes D. Garciano, Kyle Stephan S. See
  • Journal Title: IAENG International Journal of Computer Science Volume: 47 Pages: 302-309
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] 心拍変動を用いた運動中のイヌの情動評価の試み (2020)
  • Author(s): 村山 未来, 永澤 美保, 片山 真希, 池田 和司, 久保 孝富, 山川 俊貴, 藤原 幸一, 菊水 健史
  • Journal Title: 動物心理学研究 Volume: 70 Issue: 1 Pages: 15-18
  • DOI: 10.2502/janip.70.1.1
  • NAID: 130007884222
  • Peer Reviewed / Open Access
• [Journal Article] An empirical study about gait synchronization between horses (<i>Equus caballus</i>) and humans (2020)
  • Author(s): 上田江里子，久保孝富，村重哲史，永澤美保，池田和司，瀧本-猪瀬彩加
  • Journal Title: Japanese Journal of Animal Psychology Volume: 70 Issue: 2 Pages: 27-39
  • DOI: 10.2502/janip.70.2.1
  • NAID: 130007958823
  • ISSN: 0916-8419, 1880-9022
  • Peer Reviewed / Open Access
• [Journal Article] Theoretical analysis of skip connections and batch normalization from generalization and optimization perspectives (2020)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Journal Title: APSIPA Transactions on Signal and Information Processing Volume: 9 Issue: 1
  • DOI: 10.1017/atsip.2020.7
  • Peer Reviewed / Open Access
• [Journal Article] Predicting the band gap of ZnO quantum dots via supervised machine learning models (2020)
  • Author(s): Paul Rossener Regonia, Ryosuke Tani, Atsushi Ishizumi, Hisao Yanagi, Kazushi Ikeda
  • Journal Title: Optik: International Journal for Light and Electron Optics Volume: 207 Pages: 164469-164469
  • DOI: 10.1016/j.ijleo.2020.164469
  • Peer Reviewed
• [Journal Article] Data-efficient framework for personalized physiotherapy feedback (2020)
  • Author(s): Bryan Lao, Tomoya Tamei, Kazushi Ikeda
  • Journal Title: Frontiers in Computer Science Volume: 2 Pages: 3-3
  • DOI: 10.3389/fcomp.2020.00003
  • Peer Reviewed / Open Access
• [Presentation] Evaluation of the Effect of Transfer Learning to Multi-Instance Detection of Monkeys (2021)
  • Author(s): Riza Rae Pineda, Takatomi Kubo, Masaki Shimada, Kazushi Ikeda
  • Organizer: 13th Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA-ASC)
  • Int'l Joint Research
• [Presentation] Robust Tracking of Japanese Macaques from Videos in the Wild using Cascaded Networks (2021)
  • Author(s): Riza Rae Pineda, Takatomi Kubo, Masaki Shimada, Kazushi Ikeda
  • Organizer: CV4Animals: Computer Vision for Animal Behavior Tracking and Modeling Workshop in conjunction with Computer Vision and Pattern Recognition 2021 (CVPR)
  • Int'l Joint Research
• [Presentation] Quantification Analysis of Behavioral Changes after Sciatic Nerve Ligation in Rats (2020)
  • Author(s): Panyawut Sri-iesaranusorn, Saeka Shimochi, Naoaki Ono, Emrah Yatkin, Hidehiro Iida, Kazushi Ikeda, Junichiro Yoshimoto
  • Organizer: APSIPA-ASC
  • Int'l Joint Research
• [Presentation] Stochastic depthが勾配降下法の学習速度に与える影響の理論解析 (2019)
  • Author(s): 古庄康隆，池田和司
  • Organizer: 情報理論的学習理論ワークショップ
• [Presentation] ResNetとbatch normalizationによるデータ分離能力の向上 (2019)
  • Author(s): 古庄康隆，池田和司
  • Organizer: 情報論的学習理論と機械学習研究会
• [Presentation] 古典的なニューラルネットワークの問題点と加算型及び結合型ショートカットによる改善 (2019)
  • Author(s): 古庄康隆，池田和司
  • Organizer: 情報論的学習理論と機械学習研究会
• [Presentation] Fixup initializationの理論解析: 学習の高速化とResNetの汎化能力向上 (2019)
  • Author(s): 古庄康隆，池田和司
  • Organizer: 情報論的学習理論と機械学習研究会
• [Presentation] Overview: Theoretical properties of deep learning methods (2019)
  • Author(s): Kazushi Ikeda
  • Organizer: APSIPA-ASC
  • Int'l Joint Research / Invited
• [Presentation] ResNet and batch normalization improve data separability (2019)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: Asian Conference on Machine Learning
  • Int'l Joint Research / Invited
• [Presentation] Generation and visualization of tennis swing motion by conditional variational RNN with hidden Markov model (2019)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: Asian Conference on Machine Learning: Trajectory, Activity, and Behaviour workshop
  • Int'l Joint Research / Invited
• [Presentation] Theoretical analysis of the fixup initialization for fast convergence and high generalization ability (2019)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: International Conference on Machine Learning: Understanding and Improving Generalization in Deep Learning workshop
  • Int'l Joint Research / Invited
• [Presentation] Additive or concatenating skip-connections improve data separability (2019)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: International Conference on Machine Learning: Understanding and Improving Generalization in Deep Learning workshop
  • Int'l Joint Research / Invited
• [Presentation] ニューラルネットワークの隠れ層のユニット数とReLU関数が汎化能力に与える影響の理論解析 (2019)
  • Author(s): 古庄泰隆，池田和司
  • Organizer: 電子情報通信学会IBISML研究会
• [Presentation] Batch-normalizationがResNetのFIsher情報行列に与える影響の理論解析 (2019)
  • Author(s): 古庄泰隆，池田和司
  • Organizer: 電子情報通信学会IBISML研究会
• [Presentation] Effects of Skip-Connection in ResNet and Batch-Normalization on Fisher Information Matrix (2019)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: 脳と心のメカニズム冬のワークショップ
• [Presentation] ニューラルネットワークの埋め込み安定性と汎化能力の関係 (2018)
  • Author(s): 古庄泰隆，池田和司
  • Organizer: 第14回コンピューテーショナル・インテリジェンス研究会
• [Presentation] Non-asymptotic analysis of Fisher information matrices of Multi-layer perceptron, ResNet, and Batch-normalization (2018)
  • Author(s): Yasutaka Furusho, Kazushi Ikeda
  • Organizer: 6th INCF Japan Node International Workshop: Advances in Neuroinformatics
  • Int'l Joint Research