Aberrant Behavior of Pearson's Correlation Coefficient and its remedy

2021 Fiscal Year Final Research Report

• Project/Area Number: 18K03048
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Review Section: Basic Section 10020:Educational psychology-related
• Research Institution: Waseda University
• Principal Investigator: Shiina Kempei 早稲田大学, 教育・総合科学学術院, 教授 (60187317)
• Co-Investigator(Kenkyū-buntansha): 久保 沙織 東北大学, 高度教養教育・学生支援機構, 准教授 (70631943)
• Project Period (FY): 2018-04-01 – 2022-03-31
• Keywords: 相関係数 / バイアス / polychoric相関係数 / 評定尺度

Outline of Final Research Achievements

The range of Pearson's correlation coefficient r is said to be [-1,1], but we found a serious bias that the absolute value cannot attain 1 when the number of categories of the two variables are different. So we did the following:
We first thoroughly investigated the reality of the bias. The expected value of the sample correlation coefficient was robustly underestimated, and it was revealed that this bias decreased as the number of categories increased. We attempted to convey this fact to everyone who uses the correlation coefficient. We also investigated how the bias affects other statistical methods. Finally, in order to remove the bias, we proposed a new calculation method: the polychoric correlation coefficient using the EM algorithm.
We also conducted a historical study of the rating scale.

Free Research Field

計量心理学

Academic Significance and Societal Importance of the Research Achievements

相関係数は非常に多くの学問領域で統計的手法の基本要素として使用されている。本研究で明らかにした相関係数のバイアスは統計解析を危険にさらす場合があるので、相関係数を使用するすべての科学者・実務家の共通認識とならなければならないだろう。バイアスが回帰分析、因子分析のような手法にどのような影響を与えるのかをさらに調べるのが今後の課題となる。
またバイアスを除去するための方法が必要となるが、本研究で提案したEMアルゴリズムを用いたpolychoric相関係数はこの点において貢献できるであろう。