Difference between marginal likelihood and generalization error as statistical model evaluation based on algebraic geometry and structure learning theory

2019 Fiscal Year Final Research Report

• Project/Area Number: 15K00331
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Soft computing
• Research Institution: Tokyo Institute of Technology
• Principal Investigator: WATANABE SUMIO 東京工業大学, 情報理工学院, 教授 (80273118)
• Project Period (FY): 2015-04-01 – 2020-03-31
• Keywords: 情報量規準 / 交差検証 / 周辺尤度 / WAIC / WBIC

Outline of Final Research Achievements

In Bayesian statistics, two criteria, the generalization error and the marginal likelihood are well known for optimizing a model and a prior, however, their difference was not clarified.In this research, the following four results were obtained, (1) If the posterior can be approximated by a normal distribution, the optimal hyperparameters for WAIC and cross validation converge to the parameter that minimizes the average generalization error, whereas the parameter which maximizes the marginal likelihood does not. (2) Nonngeative matrix factorization is an example of singular statistical models, whose asymptotic generalization error and marginal likelihood were clarified. (3) its variational free energy was also clarified, (4) The asymptotic ditribution of the constant order term of the free energy, which is necessary to construct the most powerful test in Bayesian statistics, was clarified.

Free Research Field

情報数理

Academic Significance and Societal Importance of the Research Achievements

データの分析において統計モデルと事前分布をどのように設計したらよいかという問題は実社会においても常に必要になる課題である。この課題に対して情報量規準、交差検証、周辺尤度は、既に広く実用に用いられているが、それらの相違については必ずしも明確にはされていなかった。本研究により、次のことが明らかになった。(1)WAICと交差検証の最小化は平均汎化誤差を最小にするが、周辺尤度最大化ではそうならない。(2)非負値行列分解における汎化誤差と周辺尤度の漸近挙動の上界の値を求めることができる。(3)そのモデルの変分自由エネルギーの値を知ることができる。(4)混合正規分布のベイズ最強検定を作ることができる。