Condensation and Prediction Acceleration for Deep Learning Through Low-rank Regularization and Adaptive Proximal Methods

• 研究課題/領域番号: 23K19981
• 研究種目: 研究活動スタート支援
• 配分区分: 基金
• 審査区分: 1001:情報科学、情報工学およびその関連分野
• 研究機関: 統計数理研究所
• 研究代表者: 李 静沛 統計数理研究所, 先端データサイエンス研究系, 准教授 (00984767)
• 研究期間 (年度): 2023-08-31 – 2025-03-31
• 研究課題ステータス: 交付 (2023年度)
• 配分額: 2,860千円 (直接経費: 2,200千円、間接経費: 660千円); 2024年度: 1,430千円 (直接経費: 1,100千円、間接経費: 330千円); 2023年度: 1,430千円 (直接経費: 1,100千円、間接経費: 330千円)
• キーワード: Deep learning / Adaptive proximal method / nonlinear optimization / proximal methods / regularized optimization

研究開始時の研究の概要

This project will investigate condensing deep learning models via low-rank structures such that each layer of the neural network can be decomposed into two layers with much fewer neurons and thus fewer model parameters. We will leverage techniques from nonlinear optimization to train such models.

研究実績の概要

Our research has led to the following paper preprints:
- Huang, Zih-Syuan and Lee, Ching-pei. Regularized adaptive momentum dual averaging with an efficient inexact subproblem solver for training structured neural network, 2024. https://arxiv.org/abs/2403.14398.
- Chen, Yen-Ju, Huang, Nai-Chieh, Lee, Ching-pei, and Hsieh, Ping-Chun. Accelerated policy gradient: On the convergence rates of the Nesterov momentum for reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. https://openreview.net/forum?id=aeXRBnLoPP. Accepted.

We have also released a package based on the second paper above at the following URL: https://github.com/ismoptgroup/RAMDA

現在までの達成度 (区分)

2: おおむね順調に進展している

理由

We have finished two research papers on related directions, and released a software package based on our research outcomes. In general our progress is smooth and in an expected pace.

今後の研究の推進方策

In the coming year, the principal investigator will make some travels to attend conferences to present our research outcomes to the research community. We will also continue working on the convergence guarantees for the developed methods as well as its combination with low-rank regularization.

研究成果

• [国際共同研究] National Yang Ming Chiao Tung University/National Taiwan University(その他の国・地域 (Taiwan))
• [雑誌論文] Accelerated policy gradient: On the convergence rates of the nesterov momentum for reinforcement learning (2024)
  • 著者名/発表者名: Chen, Yen-Ju, Huang, Nai-Chieh, Lee, Ching-pei, and Hsieh, Ping-Chun
  • 雑誌名: Proceedings of the 41st International Conference on Machine Learning 巻: -
  • 査読あり / オープンアクセス / 国際共著