• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Study on learning dynamics of high-dimensional machine learning models and development of efficient learning methods

Research Project

Project/Area Number 19K20337
Research Category

Grant-in-Aid for Early-Career Scientists

Allocation TypeMulti-year Fund
Review Section Basic Section 61030:Intelligent informatics-related
Research InstitutionKyushu Institute of Technology (2021)
The University of Tokyo (2019-2020)

Principal Investigator

Nitanda Atsushi  九州工業大学, 大学院情報工学研究院, 准教授 (60838811)

Project Period (FY) 2019-04-01 – 2022-03-31
Project Status Completed (Fiscal Year 2021)
Budget Amount *help
¥4,290,000 (Direct Cost: ¥3,300,000、Indirect Cost: ¥990,000)
Fiscal Year 2021: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000)
Fiscal Year 2020: ¥1,040,000 (Direct Cost: ¥800,000、Indirect Cost: ¥240,000)
Fiscal Year 2019: ¥2,210,000 (Direct Cost: ¥1,700,000、Indirect Cost: ¥510,000)
Keywords機械学習 / 深層学習 / ニューラルネットワーク / 確率的勾配降下法 / ランジュバンダイナミクス / 確率的最適化法 / 平均場理論 / カーネル法 / 加速分散縮小法 / 超高次元ニューラルネット / 非凸最適化 / 確率的最適化
Outline of Research at the Start

深層ニューラルネットは超高次元非凸モデルであるが,種々の学習テクニックを精密に適用する事で学習が可能となり,優れたパフォーマンスを発揮する事が経験的に示されている.
しかしながら,この様な超高次元モデルが正則化無しに高い汎化性能を示す事の理論的解析は未だ発展途上である.また非常に複雑な非線形モデルであるため最適化が困難でありパラメータチューニングに多大なコストを要するという問題もある.
本研究では,超高次元モデルの学習ダイナミクスそのものが汎化性の優れたパラメータを優先的に選択する機能を備えているという考えに基き超高次元モデルの成功を裏付ける為の理論構築及び効率的学習法の開発に取り組む.

Outline of Final Research Achievements

We study learning dynamics of machine learning models, aiming to understand why high-dimensional models such as deep learning work well and to develop efficient learning methods. In particular, we obtained the following results for the (stochastic) gradient descent method, which is a representative learning method.
(1) We proved that the classification error converges exponentially under low noise conditions for classification problems using linear models. (2) We proved that the generalization ability of the two-layer neural network trained by the stochastic gradient descent method achieves optimal efficiency by refining the NTK theory. (3) We developed a way for analyzing neural networks based on the functional gradient theory of transport mapping and proposed a new learning method. (4)We developed an optimization dynamics of mean-field neural networks and proved its convergence.

Academic Significance and Societal Importance of the Research Achievements

深層学習の原理解明に向けた二種の最適化理論:NTK理論および平均場ニューラルネットワーク理論の進展に寄与した.具体的にはNTK理論を精緻化しニューラルネットワークを理論上最適な効率で学習可能であることを初めて証明し,またデータへの適応性に優れた平均場ニューラルネットワークの最適化ダイナミクスを解析する新たな研究の流れを創出した.
これらの成果は深層学習の最適化ダイナミクスの基礎を与えるもので,深層学習の効率化への重要なステップである.

Report

(4 results)
  • 2021 Annual Research Report   Final Research Report ( PDF )
  • 2020 Research-status Report
  • 2019 Research-status Report
  • Research Products

    (39 results)

All 2022 2021 2020 2019

All Journal Article (10 results) (of which Int'l Joint Research: 2 results,  Peer Reviewed: 10 results,  Open Access: 10 results) Presentation (29 results) (of which Int'l Joint Research: 11 results,  Invited: 6 results)

  • [Journal Article] Convex Analysis of the Mean Field Langevin Dynamics2022

    • Author(s)
      Atsushi Nitanda, Denny Wu, Taiji Suzuki
    • Journal Title

      Proceedings of Machine Learning Research (AISTATS2022)

      Volume: 151 Pages: 9741-9757

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] Sharp characterization of optimal minibatch size for stochastic finite sum convex optimization2021

    • Author(s)
      Nitanda Atsushi, Murata Tomoya, Suzuki Taiji
    • Journal Title

      Knowledge and Information Systems

      Volume: 63 Issue: 9 Pages: 2513-2539

    • DOI

      10.1007/s10115-021-01593-1

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space2021

    • Author(s)
      Taiji Suzuki, Atsushi Nitanda
    • Journal Title

      Advances in Neural Information Processing Systems (NeurIPS2021)

      Volume: 34 Pages: 3609-3621

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access
  • [Journal Article] Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis2021

    • Author(s)
      Atsushi Nitanda, Denny Wu, Taiji Suzuki
    • Journal Title

      Advances in Neural Information Processing Systems (NeurIPS2021)

      Volume: 34 Pages: 19608-19621

    • Related Report
      2021 Annual Research Report
    • Peer Reviewed / Open Access / Int'l Joint Research
  • [Journal Article] Exponential Convergence Rates of Classification Errors on Learning with SGD and Random Features2021

    • Author(s)
      Shingo Yashima, Atsushi Nitanda, and Taiji Suzuki
    • Journal Title

      Proceedings of Machine Learning Research (AISTATS2021)

      Volume: 130 Pages: 1954-1962

    • Related Report
      2020 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Functional Gradient Boosting for Learning Residual-like Networks with Statistical Guarantees2020

    • Author(s)
      Atsushi Nitanda and Taiji Suzuki
    • Journal Title

      Proceedings of Machine Learning Research (AISTATS2020)

      Volume: 108 Pages: 2981-2991

    • Related Report
      2020 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors2019

    • Author(s)
      Atsushi Nitanda and Taiji Suzuki
    • Journal Title

      Proceedings of Machine Learning Research (AISTATS2019)

      Volume: 89 Pages: 1417-1426

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Sharp Characterization of Optimal Minibatch Size for Stochastic Finite Sum Convex Optimization2019

    • Author(s)
      Nitanda Atsushi, Murata Tomoya, and Suzuki Taiji
    • Journal Title

      In Proceedings of IEEE International Conference on Data Mining

      Volume: - Pages: 488-497

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Data Cleansing for Models Trained with SGD2019

    • Author(s)
      Satoshi Hara, Atsushi Nitanda, and Takanori Maehara
    • Journal Title

      Advances in Neural Information Processing Systems

      Volume: 32 Pages: 4215-4224

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Open Access
  • [Journal Article] Hyperbolic Ordinal Embedding2019

    • Author(s)
      Atsushi Suzuki, Jing Wang, Feng Tian, Atsushi Nitanda, and Kenji Yamanishi
    • Journal Title

      In Proceedings of Machine Learning Research (ACML2019)

      Volume: 101 Pages: 1065-1080

    • Related Report
      2019 Research-status Report
    • Peer Reviewed / Open Access
  • [Presentation] Convex Analysis of the Mean Field Langevin Dynamics2022

    • Author(s)
      Atsushi Nitanda
    • Organizer
      Workshop on Functional Inference and Machine Intelligence
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research / Invited
  • [Presentation] Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space2021

    • Author(s)
      Taiji Suzuki, Atsushi Nitanda
    • Organizer
      Neural Information Processing Systems (NeurIPS2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis2021

    • Author(s)
      Atsushi Nitanda, Denny Wu, Taiji Suzuki
    • Organizer
      Neural Information Processing Systems (NeurIPS2021)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Convex Analysis of the Mean Field Langevin Dynamics2021

    • Author(s)
      Atsushi Nitanda, Denny Wu, Taiji Suzuki
    • Organizer
      International Conference on Artificial Intelligence and Statistics (AISTATS2022)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 平均場ニューラルネットワークの最適化法2021

    • Author(s)
      二反田篤史
    • Organizer
      日本オペレーションズ・リサーチ学会九州支部 2021年度第1回講演会・研究会
    • Related Report
      2021 Annual Research Report
    • Invited
  • [Presentation] Optimality and superiority of deep learning for estimating functions in variants of Besov spaces2021

    • Author(s)
      Taiji Suzuki, Atsushi Nitanda, Kazuma Tsuji
    • Organizer
      International Conference on Econometrics and Statistics (EcoSta)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] Fast learning rates of averaged stochastic gradient descent for over-parameterized neural networks2021

    • Author(s)
      Atsushi Nitanda, Taiji Suzuki
    • Organizer
      International Conference on Econometrics and Statistics (EcoSta)
    • Related Report
      2021 Annual Research Report
    • Int'l Joint Research
  • [Presentation] 平均場ニューラルネットワークの収束率保証付き最適化2021

    • Author(s)
      二反田篤史
    • Organizer
      日本応用数理学会年会
    • Related Report
      2021 Annual Research Report
  • [Presentation] 平均場ニューラルネットワークの効率的最適化法2021

    • Author(s)
      二反田篤史,大古一聡,Denny Wu,鈴木大慈
    • Organizer
      統計関連学会連合大会
    • Related Report
      2021 Annual Research Report
  • [Presentation] Particle Stochastic Dual Coordinate Ascent: Exponential Convergent Algorithm for Mean Field Neural Network Optimization2021

    • Author(s)
      大古一聡, 鈴木大慈, 二反田篤史, Denny Wu.
    • Organizer
      情報論的学習理論ワークショップ (IBIS)
    • Related Report
      2021 Annual Research Report
  • [Presentation] 二層ニューラルネットワークの最適化理論2021

    • Author(s)
      二反田篤史
    • Organizer
      第2回若手数学者交流会
    • Related Report
      2020 Research-status Report
  • [Presentation] When Does Preconditioning Help or Hurt Generalization?2020

    • Author(s)
      Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, and Ji Xu
    • Organizer
      International Conference on Learning Representation (ICLR2021)
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime2020

    • Author(s)
      Atsushi Nitanda and Taiji Suzuki
    • Organizer
      International Conference on Learning Representation (ICLR2021)
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] 確率的最適化法の収束解析2020

    • Author(s)
      二反田篤史
    • Organizer
      RAMP数理最適化シンポジウム
    • Related Report
      2020 Research-status Report
    • Invited
  • [Presentation] 確率的勾配降下法のNTK理論による最適収束率2020

    • Author(s)
      二反田篤史,鈴木大慈
    • Organizer
      統計関連学会連合大会
    • Related Report
      2020 Research-status Report
  • [Presentation] 粒子双対平均化法:平均場ニューラルネットワークの大域的収束保証付最適化法2020

    • Author(s)
      二反田篤史,Denny Wu, 鈴木大慈
    • Organizer
      情報論的学習理論ワークショップ (IBIS)
    • Related Report
      2020 Research-status Report
  • [Presentation] 二段階最適化によるモデル抽出攻撃に対する防御2020

    • Author(s)
      森雄人, 二反田篤史, 武田朗子
    • Organizer
      情報論的学習理論ワークショップ (IBIS)
    • Related Report
      2020 Research-status Report
  • [Presentation] When Does Preconditioning Help or Hurt Generalization?2020

    • Author(s)
      Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, and Ji Xu
    • Organizer
      The 12th OPT Workshop on Optimization for Machine Learning
    • Related Report
      2020 Research-status Report
    • Int'l Joint Research
  • [Presentation] Stochastic Gradient Descent with Exponential Convergence Rates for Classification Problems2019

    • Author(s)
      Atsushi Nitanda
    • Organizer
      Summer School 2019 on Transfer Learning
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research / Invited
  • [Presentation] 高次元ニューラルネットに対する勾配法の大域収束性と汎化性能解析2019

    • Author(s)
      二反田篤史
    • Organizer
      日本オペレーションズ・リサーチ学会 研究部会 最適化とその応用 (OPTA)
    • Related Report
      2019 Research-status Report
    • Invited
  • [Presentation] 学習アルゴリズムの大域収束性と帰納的バイアス2019

    • Author(s)
      二反田篤史
    • Organizer
      情報論的学習理論ワークショップ (IBIS)
    • Related Report
      2019 Research-status Report
    • Invited
  • [Presentation] Random Featureを用いた確率的勾配法の期待識別誤差の収束解析2019

    • Author(s)
      八嶋晋吾,二反田篤史,鈴木大慈
    • Organizer
      情報論的学習理論ワークショップ(IBIS)
    • Related Report
      2019 Research-status Report
  • [Presentation] SGDの挙動解析に基づくデータクレンジング2019

    • Author(s)
      原聡,二反田篤史,前原貴憲
    • Organizer
      情報論的学習理論ワークショップ(IBIS)
    • Related Report
      2019 Research-status Report
  • [Presentation] 高次元二層ニューラルネットに対する勾配降下法による識別誤差の大域収束性と汎化性能解析2019

    • Author(s)
      二反田篤史,鈴木大慈
    • Organizer
      情報論的学習理論ワークショップ(IBIS)
    • Related Report
      2019 Research-status Report
  • [Presentation] 識別問題に対する高次元二層ニューラルネットの大域収束性と汎化性能解析2019

    • Author(s)
      二反田篤史
    • Organizer
      情報系 WINTER FESTA Episode 5
    • Related Report
      2019 Research-status Report
  • [Presentation] Exponential convergence of stochastic gradient descent for binary classification problems2019

    • Author(s)
      Atsushi Nitanda, Taiji Suzuki
    • Organizer
      The Conference of Data Science, Statistics & Visualisation (DSSV)
    • Related Report
      2019 Research-status Report
    • Int'l Joint Research
  • [Presentation] 識別問題に対する高次元ニューラルネットの勾配降下法の大域収束性と汎化性能解析2019

    • Author(s)
      二反田篤史,鈴木大慈
    • Organizer
      日本応用数理学会年会
    • Related Report
      2019 Research-status Report
  • [Presentation] 識別問題に対する高次元二層ニューラルネットの勾配法による汎化性能解析2019

    • Author(s)
      二反田篤史,鈴木大慈
    • Organizer
      統計関連学会連合大会
    • Related Report
      2019 Research-status Report
  • [Presentation] カーネル法におけるrandom featureを用いた確率的勾配法の期待識別誤差の線形収束性2019

    • Author(s)
      八嶋晋吾,二反田篤史,鈴木大慈
    • Organizer
      統計関連学会連合大会
    • Related Report
      2019 Research-status Report

URL: 

Published: 2019-04-18   Modified: 2023-01-30  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi