Rasch calibration of Arabic reading diagnostic test items

Abstract

Diagnostic testing has always been acknowledged as a useful tool for teaching and learning. Unfortunately, the complacent belief that any given test provides diagnostic information and the laborious task of developing and managing items for diagnostic testing have adversely contributed to the scarcity of such tests in foreign language education. However, the ascendancy of autonomous and self-access language learning paradigms, coupled with advancement in computer-assisted assessment has led to a renewed interest in diagnostic language testing. Hence, the motive for this research, which was to create an initial item bank of pre-calibrated items for a computeradaptive diagnostic test of Arabic reading. Specifically, the purpose of this research was to validate an initial pool of 134 test items for diagnosing Arabic reading ability, and to calibrate the items onto one common variable, i.e. the Arabic reading comprehension. This was achieved primarily through the application of Rasch Measurement Model (RMM) to the design and validation of the test items. The design and development of the test items observed several a priori validity considerations, namely, the development and review of test specifications, text and construct mapping, and review of items. To calibrate the items onto a common scale, RMMbased vertical linking procedure with common item design and concurrent calibration were employed. The items were multiple-choice questions parceled into three different test forms of ascending difficulty, with some common items linking adjacent tests. The tests were administered to 455 non-native learners of Arabic representing three targeted samples of varying ability levels. Participants were students of International Islamic University Malaysia and Universiti Kebangsaan Malaysia. Selection of participants was done through the recommendations of Arabic program coordinators and class instructors, and at times through the snowball technique. Data were analyzed using Winsteps 3.63, a software for RMM-based analyses. The Rasch model for dichotomous data was employed for internal validation of the test items, validation of common items, and concurrent linking of the test forms. The analyses yielded 130 surviving items which were then sorted into low, average and high difficulty to constitute the initial item bank, suitable for use with computer adaptive testing. Findings indicate partial congruence between hypothesized levels of difficulty and empirical data. The research concludes that the measures in the bank were obtained from items and test responses valid for the research purpose, Arabic reading is a statistically unidimensional construct, RMM is an effective tool for designing and validating tests, and concurrent calibration is a very practical and useful procedure for vertical linking and estimation of test items. However, due to limitations of the study, no hierarchy of Arabic reading construct could yet be defined. Finally, the study offers some implications and recommendations for the use and expansion of the item bank, the development of diagnostic testing for autonomous learning, and research into the measurement construct of Arabic reading.