Construct validation of the revised listening component of the Malaysian university english test

Abstract

This paper presents the findings for content- and construct-related evidence of the newly revised version of the listening skill test that is used as part of a university entrance requirement. The Malaysian University English Test (MUET) listening component was designed and conceptualized based on Bloom’s cognitive taxonomy, encompassing 18 objectives in a 20-item instrument to gauge the mastery of listening comprehension ability. The study attempted to define the construct theoretically and operationally under logical investigation and to find the evidence of content relevance, content representation and technical quality of the items through empirical investigation. It employed the item rating procedures involving four content experts in the area of English as a Second Language Learning. Low Spearman’s rho values on inter-rater reliability indicated marked variations across the raters. Rating data were analysed using both the traditional item-objective congruence method and adjusted item index equation as the items were found to be measuring multiple objectives. Five items were rated as clearly measuring a singular valid objective while 6 were measuring multiple objectives. Of these 11 items, 9 items tested the lowest cognitive level while 2 items measured the objectives under Analysis and Synthesis. To determine construct validity, data obtained from the 250 MUET examinees were analysed using the Rasch model. Analysis of results showed that the listening construct was under-represented as the number of items was insufficient in targeting differing abilities. Two items were underfit and deleted. The study also revealed that no single format is appropriate to address all listening abilities. It is recommended that more items be added to the test. The implications of the study call for the improvement of the reliability and validity with respect to the utility of Bloom’s taxonomy in test development.