2022 Fiscal Year Final Research Report
Mathematical Foundations of Random Deep Neural Networks and their applications to machine-learning problems
| Project/Area Number |
19K20366
|
|---|
| Research Category |
Grant-in-Aid for Early-Career Scientists
|
| Allocation Type | Multi-year Fund |
|---|
| Review Section |
Basic Section 61040:Soft computing-related
|
|---|
| Research Institution | National Institute of Advanced Industrial Science and Technology |
|---|
Principal Investigator |
Karakida Ryo 国立研究開発法人産業技術総合研究所, 情報・人間工学領域, 主任研究員 (30803902)
|
|---|
| Project Period (FY) |
2019-04-01 – 2023-03-31
|
| Keywords | 深層学習 / 機械学習 / ニューラルネットワーク / 統計力学 / ランダム行列 / 数理工学 |
|---|
| Outline of Final Research Achievements |
The purpose of this research is to gain mathematical insights for deep learning, based on the analysis of random neural networks. Toward this goal, we first analyzed the eigenvalues of their Fisher information matrix, which determine the geometric structure of the parameter space. This allowed us to provide a quantitative explanation of the effects of normalization layers and appropriate settings for learning rates. In the NTK regime, characterized by learning within the range of perturbation around initial random weights, we clarified the appropriate designs of approximated natural gradient methods. Related to associative memory models, we elucidated Boltzmann machines corresponding to Modern Hopfield networks and the memory recall process in VAEs.
|
| Free Research Field |
ソフトコンピューティング
|
| Academic Significance and Societal Importance of the Research Achievements |
ランダム神経回路は古典的に理論的神経科学の枠組みで発展してきたが, 近年は深層学習にその枠組みを拡張し, たとえば逆誤差伝播における解析が進みつつある. 本研究課題もこの流れに沿うもので, 特に, 学習のプロセスに大きく影響を与えるFisher情報行列やNTK行列に着目し, 各種モデルや学習手法の性質を明らかにした点に独自性があり学術的意義がある. 本成果は様々な応用を支える基礎技術に理解を与えており, 今後の深層学習技術の研究開発を進めるうえで有用となることが期待でき, その点で社会的意義もあるといえるだろう.
|
