A RASCH analysis of reading skill across text type and item format

Abstract

Views on reading comprehension ability generally fall into two; unitary or multi-dimensional. Although the latter has garnered considerable support, it is inconclusive as regards the dimensions that make up reading ability and whether the difficulty of these reading sub-skills can be ordered hierarchically. Most studies conducted employed factor analysis to extract the number of components and to identify what make up reading ability, but the technique is unable to determine the order of difficulty of the extracted components. Additionally, the hierarchy of the sub-skills has been shown to be influenced by item characteristics. To address these issues, this research was undertaken to investigate the pattern of ordering of five commonly assessed sub-skills in L2 reading. These are understanding explicitly stated information (ESI), identifying reference (REF), deriving meaning of unknown words/ phrases (VOC), understanding the main idea of a paragraph/passage (MID), and inferring information (INF). This study also sought to determine whether the hierarchy or item ordering is similar across text type and item format. To achieve these objectives, eight sets of reading tests were put together and later administered to 668 ESL learners in matriculation and diploma programmes in two universities in Malaysia. The study employed a Rasch Partial Credit Model for data analysis as it involved both dichotomous and polytomous data. To obtain an accurate representation of hierarchy of the sub-skills, a large sample of items (n=177) were used. The common item linking technique was utilised to link the eight sets of testlets, to enable all the tests to be analysed concurrently. The findings suggest that the existence of a strictly hierarchical ordering of sub-skills cannot be fully supported. Nonetheless, there is strong evidence to suggest the presence of a hierarchy of low and high order sub-skills in reading across text type and item format. The study found that the low order sub-skills (LOS) were consistently easier than the high order sub-skills (HOS) irrespective of text type and item format. These findings substantiate the claim that different sub-skills exert different cognitive demands on readers. LOS were less cognitively demanding than HOS. In terms of the hierarchy of individual sub-skills, contrary to expectation, inference was found not to be the most difficult sub-skill. Instead, other sub-skills such as main idea were the most challenging. Another finding is that within LOS, the ordering of the two sub-skills representing LOS was consistent in all the analyses performed. By contrast, there were no consistent orderings within HOS as the pattern varied. Across item format, SAQ format in general, tends to be consistently more difficult than MCQ format in all the sub-skills be it LOS or HOS. However, when individual HOS were compared, the most difficult skill in SAQ was MID while in MCQ it was INF. Compared to text type, although LOS was distinct from HOS, the patterns of ordering of the sub-skills in HOS were less definitive. Certain HOS such as VOC, and MID tend to be more challenging in narrative texts than expository while for INF, it is just the opposite. The study has a number of implications to the field of testing of reading. One is that the importance of having varied text types when administering proficiency test to tap different language ability should not be ignored. Another implication is that item format does exert an influence on item difficulty. Thus, when designing language tests, test developers have to ensure that the item format chosen must be based on certain clear justifications.