Development of knowledge level-estimating system for multiform multiple-choice questions

2004 Fiscal Year Final Research Report Summary

• Project/Area Number: 15590464
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Single-year Grants
• Section: 一般
• Research Field: Medical sociology
• Research Institution: Jikei University School of Medicine
• Principal Investigator: KIMURA Naofumi Jikei University School of Medicine, Professor, 医学部, 教授 (80138742)
• Co-Investigator(Kenkyū-buntansha): FUKUSHIMA Osamu Jikei University School of Medicine, Professor, 医学部, 教授 (60173332)
• Project Period (FY): 2003 – 2004
• Keywords: multiple-choice questions / knowledge level / evaluation / expected correct answer probability / probability of correct answer / multiform MCQs / difficulty index / probability of incidental correct answer

Research Abstract

Multiple-choice questions(MCQs) have been extensively used in the national examinations for licensing physicians and other professionals, and this form of exam is often considered one of the most objective ways to test examinee's acquired knowledge. However, MCQs examinations have some problems of their own. One of the problems is that any question can be incidentally answered correctly at various rates, depending on the question frame (i.e.the number of choices, the number of correct answers, and the combinations of correct answers). In the case of MCQs such as choosing one or more correct answers out of multiple choices, the expected probability of a correct answer (expected P) is given as a multi-dimensional and multi-term equation for the knowledge level (q), which is the probability of discriminating correct and incorrect items. Applying this equation formula for various question types, we have developed a computer program which inversely estimates the q value (knowledge level) from the raw score rate (actual P value) in the examinations. This program is capable of treating up to three thousand multiform MCQs, which vary from choosing one correct answer out of 2-26 choices to choosing two correct answers out of 3-26 choices. To substantiate this program, we analyzed the results of authentic examinations that were used to determine college students' promotion. The analysis revealed that the mean values of the raw score rates fluctuated depending on the different types of MCQs ; however, the mean values of the estimated q values were almost constant independent of question types. Furthermore, the distribution curve of estimated q values also showed a similar pattern despite the difference in question types. This program makes it possible to estimate the actual level of examinees' knowledge, and at the same time, to evaluate the difficulty of MCQs, unaffected by their raw scores or question types.