Theoretical construction and control of deep learning based on the geometry of hierarchical models

2018 Fiscal Year Final Research Report

• Project/Area Number: 17H07390
• Research Category: Grant-in-Aid for Research Activity Start-up
• Allocation Type: Single-year Grants
• Research Field: Soft computing
• Research Institution: National Institute of Advanced Industrial Science and Technology
• Principal Investigator: Karakida Ryo 国立研究開発法人産業技術総合研究所, 情報・人間工学領域, 研究員 (30803902)
• Project Period (FY): 2017-08-25 – 2019-03-31
• Keywords: ニューラルネットワーク / 機械学習 / 深層学習 / 数理工学 / 情報幾何 / 統計力学

Outline of Final Research Achievements

This research project analyzed and developed learning algorithms in deep learning from the perspective of geometry. It led to a mathematical foundation and practical applications for deep learning without uncontrollable heuristics. In more details, we analyzed the Fisher information matrix of deep neural networks, which determines the geometry of the parameter space, and applied it to propose some gradient methods. Besides, we investigated singular regions of the parameter space caused by permutation symmetry of units. We revealed a condition where such singular region and the slowing down of training are avoidable. Moreover, we gave information geometric insight into Wasserstein distance with the entropy constraint and proposed a novel and more natural cost function based on it.

Free Research Field

ソフトコンピューティング

Academic Significance and Societal Importance of the Research Achievements

深層学習は実用ベースの開発が進み, 収束や解の性質が数理的に保証されていないヒューリスティクスへの依存が大きく, 恣意性が多く学習の制御が難しい問題がある. 本研究はこの問題に数理的な解決の基盤を与えている点で意義深い. 今後, 誰にでも使いやすい深層学習の開発につながることが期待できる. また, 学習の挙動に性能保証を与えることは, 安全性や信頼性が必要とされる実応用に深層学習を広げていくための土台となることも期待できる.