Development of Automated Essay-Scoring System for a Task-Based Writing Test

2019 Fiscal Year Final Research Report

• Project/Area Number: 16K02981
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Multi-year Fund
• Section: 一般
• Research Field: Foreign language education
• Research Institution: Meiji Gakuin University
• Principal Investigator: Sugita Yoshihito 明治学院大学, 文学部, 教授 (70363885)
• Co-Investigator(Kenkyū-buntansha): 石井 雄隆 早稲田大学, 大学総合研究センター, 助手 (90756545)
• Project Period (FY): 2016-04-01 – 2020-03-31
• Keywords: ライティング / 自動評価採点

Outline of Final Research Achievements

This study focuses on the reliability and validity of an automated essay-scoring system for a task-based writing test. The system was revised and 150 second-year high school students participated in the trial of the system and the Accuracy and Communicability values were calculated by the resulting formulas. To estimate the degree to which the indices were collectively related to the prediction of the scores, correlation analyses were conducted. The results showed that moderately high correlation existed between the scores of the both tasks and their indices. To validate the predictions of the formulas, the values were compared to the Criterion scores of the high school students. Correlations between the Accuracy and Communicability values were significant. The composite values of the two tasks were also significantly correlated. The results of the questionnaire evaluating scores of the tasks showed that the scores were acceptable and appropriate for the students.

Free Research Field

英語教育学

Academic Significance and Societal Importance of the Research Achievements

Accuracy、 Communicabilityの予測得点とCriterion スコアとの相関係数により、基準関連妥当性の検証を行ったところ比較的強い相関が見られ、また両者の合計点をTBWT によって測定されるライティング知識体系の総和 (総合評価) と位置づけて相関分析を行ったところ、強い相関が見られた。これらの結果から、自動採点結果はCriterion のパフォーマンスを一定程度反映していることが確認された。さらに、評価結果に対するアンケート調査により結果妥当性の検証を試みたところ、評価結果が利害関係者である高校生に与える影響は適切であったことが確認された。