Detecting an enemy items utilizing an NLP approach: development and evaluation for language testing

2022 Fiscal Year Final Research Report

• Project/Area Number: 20K20821
• Research Category: Grant-in-Aid for Challenging Research (Exploratory)
• Allocation Type: Multi-year Fund
• Review Section: Medium-sized Section 9:Education and related fields
• Research Institution: Nagoya University
• Principal Investigator: Mitsunaga Haruhiko 名古屋大学, 教育発達科学研究科, 准教授 (70742295)
• Project Period (FY): 2020-07-30 – 2023-03-31
• Keywords: 機械学習 / 語彙テスト / 言語テスト / 項目バンク / Automated test assembly

Outline of Final Research Achievements

When a large-scale testing program which measures examinees’ language ability is held based on an item bank, which contains various items of vocabulary, it is mandatory that the ‘enemy items’, which may be a clue to answer correctly if these items were presented at the same test administration, should not be shown on the same test forms. In this research, the method of preventing from presenting enemy items is proposed and evaluated by empirical test data. An NLP (natural language process) approach and machine learning system were adopted to estimate word similarity in each item of the item bank. Results suggested that it returns proper solutions when the item bank contains the items which measure examinees’ vocabulary, however, when items measure non-verbal ability, several external variables should be used in order to get more accurate solutions.

Free Research Field

テスト理論

Academic Significance and Societal Importance of the Research Achievements

昨今、大規模コーパスに基づいた「大規模言語モデル（large language model, LLM）」の検討が進められ、その活用が期待されているが、そのような汎用大規模言語モデルを構築するためには巨大なコーパスと膨大な計算時間が必要であり、本研究課題で検討されるような特定の用途に限定された比較的狭い範囲における応用の事例が少なかった。本研究はLLMを用いずとも、テスト事業者レベルにおいて独自のモデルを用いた敵対項目検出AIエンジンを開発することの可能性を示すものであり、コンピュータを用いたテストの自動化に寄与することが期待される。