The effect of raters and scoring rubric on the score of descriptive test.

Research Project

Project/Area Number	18K13298
Research Category	Grant-in-Aid for Early-Career Scientists
Allocation Type	Multi-year Fund
Review Section	Basic Section 10020:Educational psychology-related
Research Institution	Kyushu University
Principal Investigator	YASUNAGA Kazuhiro 九州大学, アドミッションセンター, 准教授 (80777665)
Project Period (FY)	2018-04-01 – 2021-03-31
Project Status	Completed (Fiscal Year 2020)
Budget Amount *help	¥4,160,000 (Direct Cost: ¥3,200,000、Indirect Cost: ¥960,000) Fiscal Year 2020: ¥1,300,000 (Direct Cost: ¥1,000,000、Indirect Cost: ¥300,000) Fiscal Year 2019: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000) Fiscal Year 2018: ¥1,430,000 (Direct Cost: ¥1,100,000、Indirect Cost: ¥330,000)
Keywords	国語テスト / 大学入試 / 記述式問題 / 評定者 / 評価基準 / 一致度 / 字数制限 / 項目分析 / 回答欄 / 解答類型 / 記述式テスト
Outline of Final Research Achievements	This study examines if variations in the scoring rubric and raters for a descriptive item affect test-takers’ evaluations in a Japanese language comprehension test. The test was administered to about 300 students in the second year of high school. Two types of scoring rubric were used. Rubric A provides the correct answer, the incorrect answer, and no-response. Rubric B had a gradual evaluation in addition to the contents of rubric A. The items were evaluated based on rubrics A and B. Therefore, rubric B, having a gradual evaluation, allowed us to comprehend which statements were missing compared to the correct answer. Moreover, rubric B increased the discrimination value in a certain item. Regarding the raters, the degree of agreement between the three raters’ evaluations was examined. Consequently, the level of agreement was moderate for items in which the decision to evaluate was left to the raters, suggesting that the evaluation could vary greatly depending on the raters.
Academic Significance and Societal Importance of the Research Achievements	本研究成果の学術的意義は，わが国においてほとんど検討されてこなかった国語テストにおける記述式問題を対象に，評価基準や評定者がテストの結果に及ぼす影響を明らかにした点である。また，教育現場の協力により得られた成果であるため，実際に教育へ応用できる点が本研究成果の社会的意義である。