2022 Fiscal Year Final Research Report
Creating Auxiliary Questions for Explainable Evaluation of Machine Reading Comprehension
Project/Area Number |
20K23335
|
Research Category |
Grant-in-Aid for Research Activity Start-up
|
Allocation Type | Multi-year Fund |
Review Section |
1001:Information science, computer engineering, and related fields
|
Research Institution | National Institute of Informatics |
Principal Investigator |
Sugawara Saku 国立情報学研究所, コンテンツ科学研究系, 助教 (10855894)
|
Project Period (FY) |
2020-09-11 – 2023-03-31
|
Keywords | 自然言語処理 / 計算言語学 / 自然言語理解 |
Outline of Final Research Achievements |
Developing natural language understanding systems requires detailed analysis and evaluation of the language understanding process. However, existing tasks have not ensured sufficient accountability for systems' capabilities. This study focused on reading comprehension questions and constructed a new dataset that enables detailed evaluation by testing the understanding of the rationale in the question answering process. We used crowdsourcing to collect rationale texts for the correct and incorrect answers of existing multiple-choice reading comprehension questions, and then used the rationale information to create an auxiliary set of multiple-choice questions that help us to determine whether or not a system correctly answers the question, including the rationale in a consistent manner.
|
Free Research Field |
自然言語処理
|
Academic Significance and Societal Importance of the Research Achievements |
言語理解を実現するシステムの構築は自然言語処理における最大の目標のひとつである。システムを着実に開発するには言語理解に関する精緻な分析と評価が必要であり、本研究によって得られたデータセットは読解問題の回答に至るまでのプロセスを分解して補助的な問題として課すことで詳細な評価を可能にした。これにより現状のシステムの限界が示され、本データセットは今後の改善を促進する上で重要な役割を果たす。
|