Using computer analyses to improve student writing outcomes

Abstract

This research endeavoured to investigate the effectiveness of teaching writing to 52 ESL students using lesson content based on computer analyses. The effectiveness of the teaching was determined by examining the changes in 76 language features analysed by Coh-Metrix 3.0, and changes in holistic scores and component scores obtained through the ESL Composition Profile. Coh-Metrix was also used to analyse eight well-written essays to create a Genre Target Profile which referenced Coh-Metrix norms for Language Arts. The subjects were required to write an essay following each of four lessons. These essays were analysed to determine areas of strength and weakness. This data was then compared with the Genre Target Profile and grade level equivalents showed whether measures of individual language features were too low or too high. Lesson content for the control group strictly followed the order of the course materials, whereas content for each treatment group lesson was based on Coh-Metrix analyses of the essays submitted following the previous lesson. A fifth lesson was used to return the final essay, answer questions and teach content requested by the subjects. Paired-sample t-tests were used to determine the significance of changes from one essay to the next and between Essay 1 and Essay 4. The findings show that changes in language features for both groups fluctuated from essay to essay. Changes in language features between Essay 1 and Essay 4 show that the treatment group improved in 48 features, of which four also showed a statistically significant improvement. In contrast, the control group improved in only 38 features; improvements in three of these were statistically significant. Positive changes were also seen in both groups’ overall writing proficiency according to the ESL Composition Profile. Additionally, the treatment group slightly outperformed the control group with regard to the number of positive changes in component scores of the ESL Composition Profile. The findings of this research indicate that Coh-Metrix is a tool that can assist classroom teachers create profiles of their students and identify areas of strength and weakness in order to prepare more effective lessons.