Automated essay scoring using item response theory that considers rubric characteristics

2021 Fiscal Year Final Research Report

• Project/Area Number: 19K21751
• Research Category: Grant-in-Aid for Challenging Research (Exploratory)
• Allocation Type: Multi-year Fund
• Review Section: Medium-sized Section 9:Education and related fields
• Research Institution: The University of Electro-Communications
• Principal Investigator: Ueno Maomi 電気通信大学, 大学院情報理工学研究科, 教授 (50262316)
• Co-Investigator(Kenkyū-buntansha): 宇都 雅輝 電気通信大学, 大学院情報理工学研究科, 准教授 (10732571) ; 宮澤 芳光 独立行政法人大学入試センター, 研究開発部, 助教 (70726166)
• Project Period (FY): 2019-06-28 – 2022-03-31
• Keywords: 自動採点 / ルーブリック / 項目反応理論

Outline of Final Research Achievements

In automated essay scoring (AES), essays are automatically graded without human raters. Many AES models based on various manually designed features or various architectures of deep neural networks have been proposed over the past few decades. Each AES model has unique advantages and characteristics. Therefore, rather than using a single AES model, appropriate integration of predictions from various AES models is expected to achieve higher scoring accuracy. In the present paper, we develops 1) a new item response theory model that can estimate scores while considering characteristics of individual human-raters and rubric-items, and 2) a method that uses the item response theory model to integrate prediction scores from various AES models while taking into account differences in the characteristics of scoring behavior.

Free Research Field

教育工学

Academic Significance and Societal Importance of the Research Achievements

既存の自動採点機を一つ一つ深く研究しても現実の小論文自動採点を実現することは難しい．それに対して提案手法は一つの自動採点機を単独で用いるよりも必ず精度を向上させるという利点があり，急務である小論文の自動採点の実用化を大幅に加速すると期待できる．さらに，このアプローチは個々の自動採点機を統合して人間を近似する手法の研究のみに集中できることや，新しく精度の良い自動採点機が提案されても簡単に組み入れることができる，など長期的な利点が多い．自動採点が実用化されれば大学入試への応用だけでなく，高度な支援機能を持つ学習支援システムなどが実現でき，教育界・産業界にもインパクトが期待できる.