Study on learning dynamics of high-dimensional machine learning models and development of efficient learning methods

2021 Fiscal Year Final Research Report

• Project/Area Number: 19K20337
• Research Category: Grant-in-Aid for Early-Career Scientists
• Allocation Type: Multi-year Fund
• Review Section: Basic Section 61030:Intelligent informatics-related
• Research Institution: Kyushu Institute of Technology (2021); The University of Tokyo (2019-2020)
• Principal Investigator: Nitanda Atsushi 九州工業大学, 大学院情報工学研究院, 准教授 (60838811)
• Project Period (FY): 2019-04-01 – 2022-03-31
• Keywords: 機械学習 / 深層学習 / ニューラルネットワーク / 確率的勾配降下法 / ランジュバンダイナミクス

Outline of Final Research Achievements

We study learning dynamics of machine learning models, aiming to understand why high-dimensional models such as deep learning work well and to develop efficient learning methods. In particular, we obtained the following results for the (stochastic) gradient descent method, which is a representative learning method.
(1) We proved that the classification error converges exponentially under low noise conditions for classification problems using linear models. (2) We proved that the generalization ability of the two-layer neural network trained by the stochastic gradient descent method achieves optimal efficiency by refining the NTK theory. (3) We developed a way for analyzing neural networks based on the functional gradient theory of transport mapping and proposed a new learning method. (4)We developed an optimization dynamics of mean-field neural networks and proved its convergence.

Free Research Field

機械学習

Academic Significance and Societal Importance of the Research Achievements

深層学習の原理解明に向けた二種の最適化理論：NTK理論および平均場ニューラルネットワーク理論の進展に寄与した．具体的にはNTK理論を精緻化しニューラルネットワークを理論上最適な効率で学習可能であることを初めて証明し，またデータへの適応性に優れた平均場ニューラルネットワークの最適化ダイナミクスを解析する新たな研究の流れを創出した．
これらの成果は深層学習の最適化ダイナミクスの基礎を与えるもので，深層学習の効率化への重要なステップである．