• Search Research Projects
  • Search Researchers
  • How to Use
  1. Back to previous page

Difference between marginal likelihood and generalization error as statistical model evaluation based on algebraic geometry and structure learning theory

Research Project

Project/Area Number 15K00331
Research Category

Grant-in-Aid for Scientific Research (C)

Allocation TypeMulti-year Fund
Section一般
Research Field Soft computing
Research InstitutionTokyo Institute of Technology

Principal Investigator

WATANABE SUMIO  東京工業大学, 情報理工学院, 教授 (80273118)

Project Period (FY) 2015-04-01 – 2020-03-31
Project Status Completed (Fiscal Year 2019)
Budget Amount *help
¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000)
Fiscal Year 2019: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2018: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2017: ¥910,000 (Direct Cost: ¥700,000、Indirect Cost: ¥210,000)
Fiscal Year 2016: ¥780,000 (Direct Cost: ¥600,000、Indirect Cost: ¥180,000)
Fiscal Year 2015: ¥910,000 (Direct Cost: ¥700,000、Indirect Cost: ¥210,000)
Keywords情報量規準 / 交差検証 / 周辺尤度 / WAIC / WBIC / 汎化誤差 / 自由エネルギー / 実対数閾値 / 漸近挙動 / ゼータ関数 / クロスバリデーション / 汎化損失 / 双有理不変量 / 交差損失 / 情報量規準WAIC / 非負値行列分解 / 汎関数分散 / 代数幾何
Outline of Final Research Achievements

In Bayesian statistics, two criteria, the generalization error and the marginal likelihood are well known for optimizing a model and a prior, however, their difference was not clarified.In this research, the following four results were obtained, (1) If the posterior can be approximated by a normal distribution, the optimal hyperparameters for WAIC and cross validation converge to the parameter that minimizes the average generalization error, whereas the parameter which maximizes the marginal likelihood does not. (2) Nonngeative matrix factorization is an example of singular statistical models, whose asymptotic generalization error and marginal likelihood were clarified. (3) its variational free energy was also clarified, (4) The asymptotic ditribution of the constant order term of the free energy, which is necessary to construct the most powerful test in Bayesian statistics, was clarified.

Academic Significance and Societal Importance of the Research Achievements

データの分析において統計モデルと事前分布をどのように設計したらよいかという問題は実社会においても常に必要になる課題である。この課題に対して情報量規準、交差検証、周辺尤度は、既に広く実用に用いられているが、それらの相違については必ずしも明確にはされていなかった。本研究により、次のことが明らかになった。(1)WAICと交差検証の最小化は平均汎化誤差を最小にするが、周辺尤度最大化ではそうならない。(2)非負値行列分解における汎化誤差と周辺尤度の漸近挙動の上界の値を求めることができる。(3)そのモデルの変分自由エネルギーの値を知ることができる。(4)混合正規分布のベイズ最強検定を作ることができる。

Report

(6 results)
  • 2019 Annual Research Report   Final Research Report ( PDF )
  • 2018 Research-status Report
  • 2017 Research-status Report
  • 2016 Research-status Report
  • 2015 Research-status Report
  • Research Products

    (26 results)

All 2020 2019 2018 2017 2016 2015 Other

All Journal Article (9 results) (of which Peer Reviewed: 5 results) Presentation (15 results) (of which Int'l Joint Research: 4 results,  Invited: 6 results) Book (1 results) Remarks (1 results)

  • [Journal Article] Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization.2020

    • Author(s)
      Naoki Hayashi, Sumio Watanabe.
    • Journal Title

      SN Computer Science

      Volume: 1 Issue: 2 Pages: 1-22

    • DOI

      10.1007/s42979-020-0071-3

    • Related Report
      2019 Annual Research Report
    • Peer Reviewed
  • [Journal Article] 混合多項分布のベイズ汎化誤差の漸近挙動2020

    • Author(s)
      渡邉匠、渡辺澄夫
    • Journal Title

      信学技報

      Volume: IBISML2019-18 Pages: 1-8

    • Related Report
      2019 Annual Research Report
  • [Journal Article] Swish 関数を用いた階層型神経回路網の実対数閾値2020

    • Author(s)
      田中来輝、渡辺澄夫
    • Journal Title

      信学技報

      Volume: IBISML2019-18 Pages: 9-5

    • Related Report
      2019 Annual Research Report
  • [Journal Article] 最適な確率分布が一意でないときのベイズ学習曲線2020

    • Author(s)
      永安修也、渡辺澄夫
    • Journal Title

      信学技報

      Volume: NC2019-14 Pages: 107-112

    • Related Report
      2019 Annual Research Report
  • [Journal Article] Higher Order Equivalence of Bayes Cross Validation and WAIC2018

    • Author(s)
      Sumio Watanabe
    • Journal Title

      Springer Proceedings in Mathematics and Statistics

      Volume: 25210.1007/978-3-319-97798-0 Pages: 47-73

    • Related Report
      2018 Research-status Report
    • Peer Reviewed
  • [Journal Article] Upper bound of Bayesian generalization error in non-negative matrix factorization2017

    • Author(s)
      Hayashi Naoki、Watanabe Sumio
    • Journal Title

      Neurocomputing

      Volume: 266 Pages: 21-28

    • DOI

      10.1016/j.neucom.2017.04.068

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] Phase Transition Structure of Variational Bayesian Nonnegative Matrix Factorization2017

    • Author(s)
      Kohjima Masahiro、Watanabe Sumio
    • Journal Title

      Proc. of ICANN

      Volume: Vol.2 Pages: 146-154

    • DOI

      10.1007/978-3-319-68612-7_17

    • ISBN
      9783319686110, 9783319686127
    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] Tighter upper bound of real log canonical threshold of non-negative matrix factorization and its application to Bayesian inference2017

    • Author(s)
      Hayashi Naoki、Watanabe Sumio
    • Journal Title

      IEEE Symposium Series on Computational Intelligence

      Volume: Vol.1 Pages: 1-8

    • DOI

      10.1109/ssci.2017.8280811

    • Related Report
      2017 Research-status Report
    • Peer Reviewed
  • [Journal Article] Difference between Bayes Cross Validation and WAIC for Conditional Independent Samples2017

    • Author(s)
      Sumio Watanabe
    • Journal Title

      Proc. of 10th WITMSE2017

      Volume: Vol.1 Pages: 38-41

    • Related Report
      2017 Research-status Report
  • [Presentation] 神経回路網研究における代数幾何学的方法とその成果について2019

    • Author(s)
      渡辺澄夫
    • Organizer
      神経回路学会
    • Related Report
      2019 Annual Research Report
    • Invited
  • [Presentation] 混合正規分布の均一性検定におけるベイズ検定統計量の漸近挙動2018

    • Author(s)
      仮屋夏樹 渡辺澄夫
    • Organizer
      2018年度統計関連連合大会
    • Related Report
      2018 Research-status Report
  • [Presentation] Cross Validation and WAIC in layered neural networks2018

    • Author(s)
      Sumio Watanabe
    • Organizer
      Deep Learning: Theory, Algorithms, and Applications
    • Related Report
      2017 Research-status Report
    • Int'l Joint Research / Invited
  • [Presentation] 混合ポアソン分布の実対数閾値とベイズ汎化誤差について、2018

    • Author(s)
      佐藤件一郎、渡辺澄夫、
    • Organizer
      情報論的学習理論と機械学習研究会
    • Related Report
      2017 Research-status Report
  • [Presentation] 確率行列分解の実対数閾値とBayes学習への応用、2018

    • Author(s)
      林 直輝、渡辺澄夫、
    • Organizer
      情報論的学習理論と機械学習研究会
    • Related Report
      2017 Research-status Report
  • [Presentation] 非負値行列分解における実対数閾値の実験的考察2017

    • Author(s)
      林直輝 渡辺澄夫
    • Organizer
      電子情報通信学会NC研究会
    • Place of Presentation
      機械振興会館
    • Year and Date
      2017-03-13
    • Related Report
      2016 Research-status Report
  • [Presentation] Bayesian sparse regression and information criteria for singular models. International Workshop2017

    • Author(s)
      渡邊澄夫
    • Organizer
      Algebraic and Geometric Methods in Statistics
    • Place of Presentation
      統計数理研究所
    • Year and Date
      2017-03-07
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research
  • [Presentation] 非負値行列分解の実対数閾値とBayes学習への応用2016

    • Author(s)
      林直輝 渡辺澄夫
    • Organizer
      電子情報通信学会情報論的学習理論研究会
    • Place of Presentation
      京都大学
    • Year and Date
      2016-11-16
    • Related Report
      2016 Research-status Report
  • [Presentation] ベイズ推論2016

    • Author(s)
      渡邊澄夫
    • Organizer
      電子情報通信学会ソサイエティ大会
    • Place of Presentation
      北海道大学
    • Year and Date
      2016-09-20
    • Related Report
      2016 Research-status Report
    • Invited
  • [Presentation] Higher Order Analysis of Bayesian Cross Validation in Regular Asymptotic Theory2016

    • Author(s)
      渡邊澄夫
    • Organizer
      Information Geometry and its Applications IV
    • Place of Presentation
      Liblice Castle Hote, Czech Republic
    • Year and Date
      2016-06-12
    • Related Report
      2016 Research-status Report
    • Int'l Joint Research / Invited
  • [Presentation] 非ガウス可解モデルを用いたレプリカモンテカルロ法 による自由エネルギーの計算精度の評価2016

    • Author(s)
      須貝将士 渡邊澄夫
    • Organizer
      信学技報
    • Place of Presentation
      玉川大学
    • Year and Date
      2016-03-23
    • Related Report
      2015 Research-status Report
  • [Presentation] ハミルトニアンモンテカルロ法を用いた神経回路網の学習と汎化誤差の推定について2016

    • Author(s)
      中村文士 渡邊澄夫
    • Organizer
      信学技報
    • Place of Presentation
      統計数理研究所
    • Year and Date
      2016-03-17
    • Related Report
      2015 Research-status Report
  • [Presentation] 潜在変数を持つモデルの評価について2016

    • Author(s)
      渡邊澄夫
    • Organizer
      データ科学シンポジウム
    • Place of Presentation
      大阪大学
    • Year and Date
      2016-01-22
    • Related Report
      2015 Research-status Report
    • Invited
  • [Presentation] Cross Validation and WAIC in Statistical Model Evaluation2016

    • Author(s)
      渡邊澄夫
    • Organizer
      Symposium ‐ What is a good model?
    • Place of Presentation
      統計数理研究所
    • Year and Date
      2016-01-12
    • Related Report
      2015 Research-status Report
    • Int'l Joint Research / Invited
  • [Presentation] スパース表現を用いた非線形多層主成分分析における 学習結果の分類法について2015

    • Author(s)
      香田夏輝 渡邊澄夫
    • Organizer
      情報論的学習理論と機械学習研究会
    • Place of Presentation
      つくば国際会議場
    • Year and Date
      2015-11-16
    • Related Report
      2015 Research-status Report
  • [Book] Mathematical theory of Bayesian statistics2018

    • Author(s)
      Sumio Watanabe
    • Total Pages
      300
    • Publisher
      CRC Press
    • ISBN
      9781482238068
    • Related Report
      2018 Research-status Report
  • [Remarks] 渡辺澄夫

    • URL

      http://watanabe-www.math.dis.titech.ac.jp/users/swatanab/index-j.html

    • Related Report
      2019 Annual Research Report 2018 Research-status Report 2017 Research-status Report 2016 Research-status Report 2015 Research-status Report

URL: 

Published: 2015-04-16   Modified: 2021-02-19  

Information User Guide FAQ News Terms of Use Attribution of KAKENHI

Powered by NII kakenhi