Advance of artificial intelligence by theoretical investigation of deep learning

• Project/Area Number: 18K19793
• Research Category: Grant-in-Aid for Challenging Research (Exploratory)
• Allocation Type: Multi-year Fund
• Review Section: Medium-sized Section 60:Information science, computer engineering, and related fields
• Research Institution: The Institute of Statistical Mathematics
• Principal Investigator: Fukumizu Kenji 統計数理研究所, 数理・推論研究系, 教授 (60311362)
• Co-Investigator(Kenkyū-buntansha): 鈴木 大慈 東京大学, 大学院情報理工学系研究科, 准教授 (60551372) ; 今泉 允聡 東京大学, 大学院総合文化研究科, 准教授 (90814088)
• Project Period (FY): 2018-06-29 – 2021-03-31
• Project Status: Completed (Fiscal Year 2020)
• Budget Amount: ¥6,370,000 (Direct Cost: ¥4,900,000、Indirect Cost: ¥1,470,000); Fiscal Year 2020: ¥2,080,000 (Direct Cost: ¥1,600,000、Indirect Cost: ¥480,000); Fiscal Year 2019: ¥2,080,000 (Direct Cost: ¥1,600,000、Indirect Cost: ¥480,000); Fiscal Year 2018: ¥2,210,000 (Direct Cost: ¥1,700,000、Indirect Cost: ¥510,000)
• Keywords: 人工知能 / 深層学習 / 機械学習 / ニューラルネットワーク / 理論解析

Outline of Final Research Achievements

This study theoretically analyzes the learning dynamics of deep neural networks with mathematical approaches. We have obtained the following results; (1) In estimating non-smooth functions, the deep neural networks have advantages over conventional models with fixed bases, (2) The sdalle point structure has been revelaed in the case that a network has surplus hidden units (3) Sufficient conditions have been obtained for the stability of deep generative models. Also, we have developed a meta-learing method for estimating causal directions with a small number of data.

Academic Significance and Societal Importance of the Research Achievements

深層学習は応用面からの成功により現在の人工知能の基盤技術となっているが，モデルが強い非線形性を持つことから理論的な解析を行うことは容易ではなくブラックボックスとして使われる場合が多い．本研究は数理的手法で深層学習の性質を理論的に明らかにするものであり，ブラックボックスを超えた深層学習の理論，特に学習によって得られたネットワークの信頼性や，学習アルゴリズムの安定性に関して重要な知見が得られた．

Research Products

• [Int'l Joint Research] Stanford University(米国)
• [Int'l Joint Research] Cornell University(米国)
• [Journal Article] A General Class of Transfer Learning Regression without Implementation Cost (2021)
  • Author(s): Shunya Minami, Song Liu, Stephen Wu, Kenji Fukumizu, Ryo Yoshida
  • Journal Title: Proceedings of 35th AAAI Conference on Artificial Intelligence Volume: 1 Pages: 1-6
  • Peer Reviewed / Open Access
• [Journal Article] Meta Learning for Causal Direction (2021)
  • Author(s): Jean-Francois Ton, Dino Sejdinovic, Kenji Fukumizu
  • Journal Title: Proceedings of 35th AAAI Conference on Artificial Intelligence Volume: 1 Pages: 1-6
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Exchangeable Deep Neural Networks for Set-to-Set Matching and Learning (2020)
  • Author(s): Saito, Y., Nakamura, T., Hachiya, H. and Fukumizu, K.
  • Journal Title: Computer Vision -- ECCV 2020 Volume: 1 Pages: 626-646
  • DOI: 10.1007/978-3-030-58520-4_37
  • ISBN: 9783030585198, 9783030585204
  • Peer Reviewed / Open Access
• [Journal Article] Smoothness and Stability in GANs (2020)
  • Author(s): Chu, C. Minami, K. and Fukumizu, K.
  • Journal Title: International Conferences on Learning Representations 2020 Volume: 1 Pages: 1-8
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] On Random Subsampling of Gaussian Process Regression: A Graphon-Based Analysis (2020)
  • Author(s): Kohei.Hayashi. Masaaki.Imaizumi. Yuichi Yoshida
  • Journal Title: Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR Volume: 108 Pages: 2055-2065
  • Peer Reviewed / Open Access
• [Journal Article] Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality (2020)
  • Author(s): Ryumei Nakada. Masaaki Imaizumi
  • Journal Title: Journal of Machine Learning Research Volume: 21 Pages: 1-38
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Smoothness and Stability in GANs (2020)
  • Author(s): C. Chu, K. Minami, and K. Fukumzu
  • Journal Title: International Conference on Representation Learing Volume: 1 Pages: 1-31
  • Peer Reviewed / Open Access / Int'l Joint Research
• [Journal Article] Semi-flat minima and saddle points by embedding neural networks to overparameterization (2019)
  • Author(s): K. Fukumizu, S. Yamaguchi, Y. Mototake, and M. Tanaka
  • Journal Title: Advances in Neural Processing Systems Volume: 32 Pages: 13868-13876
  • Peer Reviewed / Open Access
• [Journal Article] A simple method to construct confidence bands in functional linear regression (2019)
  • Author(s): Masaaki Imaizumi and Kengo Kato
  • Journal Title: Statistica Sinica Volume: *
  • Peer Reviewed / Int'l Joint Research
• [Journal Article] Statistically Efficient Estimation for Non-Smooth Probability Densities (2018)
  • Author(s): Masaaki Imaizumi, Takanori Maehara, Yuichi Yoshida
  • Journal Title: Proceedings of Machine Learning Research Workshop & Conference Proceedings (AISTATS 2018) Volume: 84 Pages: 978-987
  • Peer Reviewed
• [Journal Article] Deep Neural Networks Learn Non-Smooth Functions Effectively (2018)
  • Author(s): 今泉允聡、福水 健次
  • Journal Title: 2018年度統計関連学会連合大会講演予稿集 Volume: ?
• [Journal Article] Functional gradient boosting based on residual network perception (2018)
  • Author(s): Atsushi Nitanda and Taiji Suzuki
  • Journal Title: Proceedings of the 35th International Conference on Machine Learning: PMLR Volume: 80:
  • Peer Reviewed / Open Access
• [Journal Article] Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models (2018)
  • Author(s): Atsushi Nitanda and Taiji Suzuki
  • Journal Title: AISTATS2018, Proceedings of Machine Learning Research Volume: 84
  • Peer Reviewed / Open Access
• [Journal Article] Fast generalization error bound of deep learning from a kernel perspective (2018)
  • Author(s): Taiji Suzuki
  • Journal Title: AISTATS2018, Proceedings of Machine Learning Research Volume: 84
  • Peer Reviewed / Open Access
• [Presentation] Generalization Analysis of Deep Models with Loss Surface and Likelihood Models (2021)
  • Author(s): Masaaki Imaizumi
  • Organizer: Workshop on Functional Inference and Machine Intelligence 2021
  • Int'l Joint Research
• [Presentation] 深層生成モデルの理論と応用 (2020)
  • Author(s): 福水健次
  • Organizer: 第5回 統計・機械学習若手シンポジウム
  • Invited
• [Presentation] Smoothness and Stability in Learning Generative Adversarial Networks (2020)
  • Author(s): Kenji Fukumizu
  • Organizer: RIKEN AIP Mathematical Seminar
  • Invited
• [Presentation] Generalization Analysis for Mechanism of Deep Learning by Statistics and Learning Theory (2020)
  • Author(s): Masaaki Imaizumi
  • Organizer: Seoul National University Seminar
  • Invited
• [Presentation] 深層学習の原理の理解に向けた理論の試み (2020)
  • Author(s): 今泉允聡
  • Organizer: 諸科学における大規模データと統計数理モデリング
  • Invited
• [Presentation] Fast convergence on perfect classification for functional data (2020)
  • Author(s): Tomoya Wakayama. Masaaki Imaizumi
  • Organizer: Computational and Methodological Statistics
  • Int'l Joint Research / Invited
• [Presentation] Smoothness and Stability in Learning GANs (2019)
  • Author(s): Kenji Fukumizu
  • Organizer: Workshop on Functional Inference and Machine Intelligence
  • Invited
• [Presentation] Generalization error of deep learning with connection to sparse estimation in function space (2019)
  • Author(s): Taiji Suzuki:
  • Organizer: Workshop on Functional Inference and Machine Intelligence
  • Int'l Joint Research / Invited
• [Presentation] Compressing deep neural network and its generalization error analysis via kernel theory (2019)
  • Author(s): Taiji Suzuki
  • Organizer: Reinforcement Learning & Biological Intelligence, learning from biology, learning for biology
  • Int'l Joint Research / Invited
• [Presentation] Deep Neural Networks Learn Non-Smooth Functions Effectively (2018)
  • Author(s): Masaaki Imaizumi and Kenji Fukumizu
  • Organizer: ICML 2018 Workshop on Theory of Deep Learning
  • Int'l Joint Research
• [Presentation] 汎化誤差評価によるGANの理論解析 (2018)
  • Author(s): 今泉 允聡、福水 健次
  • Organizer: 第21回情報論的学習理論ワークショップ (IBIS 2018)
  • Invited
• [Presentation] Adaptivity of Deep ReLU Network for Learning in Besov Spaces. (2018)
  • Author(s): Taiji Suzuki
  • Organizer: Forum "Math-for-Industry" 2018 - Big Data Analysis, AI, Fintech, Math in Finances and Economics
  • Int'l Joint Research / Invited
• [Remarks] Kenji Fukumizu's home page
  • URL: http://www.ism.ac.jp/~fukumizu/