The Subjective and Objective Interface of Bias Detection on language tests

2004 Fiscal Year Final Research Report Summary

• Project/Area Number: 15520366
• Research Category: Grant-in-Aid for Scientific Research (C)
• Allocation Type: Single-year Grants
• Section: 一般
• Research Field: Foreign language education
• Research Institution: Aichi Prefectural University
• Principal Investigator: OKABE Junko Aichi Prefectural University, Faculty of Language, Associate Professor, 文学部, 助教授 (20295570)
• Co-Investigator(Kenkyū-buntansha): ROSS Steven Kwansei Gakuin University, School of Policy Studies, Professor, 大学院・総合政策学部, 教授 (20278782)
• Project Period (FY): 2003 – 2004
• Keywords: gender bias / test / DIF / schemata / subjectivity

Research Abstract

Test validity is predicted on there being a lack of bias in tasks, items, or test content. It is well known that factors such as test candidates' mother tongue, life experiences and socialization practices of the wider community may serve to inject subtle interactions between individuals' background and the test content. When the gender of the test candidate interacts further with these factors, the potential for item bias to influence test performances grows. A dilemma faced by test designers concerns how they can proactively screen test content for possible sources of bias. Conventional practices in many contexts rely on the subjective opinion of review panels in detecting potentially biased items. In the last two decades, this practice has been rivaled by the increased availability of item bias diagnostic software. Few studies have compared the relative accuracy and cost utility of the two approaches. The present study makes just that comparison. A fourpassage, 20 item reading comprehension test was given to a stratified sample of 825 high school students and college undergraduates at five Japanese institutions. The sampling included a focus group of 468 female students compared to a reference group of 357 male English as a foreign language learners. The test passages and items were also given to a panel of 97 in-service and pre-service EFL teachers for subjective ratings of potential gender bias. The results of the actual item responses were then empirically checked for evidence of ditl+erential item functioning using Simultaneous Item Bias analysis, the Mantel-Haenszel Delta method, and logistic regression. Concordance analyses of the subjective and objective methods suggest that subjective screening of bias overestimates the extent of actual item bias. Implications for cost-effective approaches to item bias detection are discussed.