Item response theory for performance assessment and its applications

2020 Fiscal Year Final Research Report

• Project/Area Number: 17H04726
• Research Category: Grant-in-Aid for Young Scientists (A)
• Allocation Type: Single-year Grants
• Research Field: Educational technology
• Research Institution: The University of Electro-Communications
• Principal Investigator: Masaki Uto 電気通信大学, 大学院情報理工学研究科, 准教授 (10732571)
• Project Period (FY): 2017-04-01 – 2021-03-31
• Keywords: パフォーマンス評価 / 項目反応理論 / eテスティング / 統計的自然言語処理

Outline of Final Research Achievements

In performance assessment where human raters subjectively grade examinees’ performances, it is important to estimate examinee ability while removing effects of raters’ biases. The purpose of this study is to develop and evaluate item response theory (IRT) models for performance assessment that can estimate examinee ability while removing rater bias effects. Concretely, we conducted the following three studies. 1) Development of a new IRT model incorporating various rater characteristic parameters to improve robustness against aberrant raters. 2) Development of an efficient Markov chain Monte Carlo method using the No-U-Turn sampler algorithm for the proposed IRT model. 3) Extensions and applications of the proposed method for various performance assessment situations.

Free Research Field

教育工学

Academic Significance and Societal Importance of the Research Achievements

本研究で開発する技術は，記述・論述式試験や語学試験，実技試験をはじめ，オンライン学習環境における学習者同士の相互評価や入社試験・人事考課で行われる面接試験など，様々なパフォーマンス評価に活用できる．本技術は，パフォーマンス評価の基礎理論として広く活用される可能性が高く，その学術的・社会的インパクトは大きい．また，本研究の適用範囲は，教育評価分野に限定されない．本技術は，オンラインショップにおける商品のレイティングやクラウドソーシングの品質評価など，評価者を伴う様々な評価データに広く適用可能であり，様々な応用タスクの性能向上に寄与すると期待できる．