Effects of Technology Modes on Ratings of Learner Recordings

Abstract

While research has investigated the effect of visuals in tests of listening
comprehension (e.g., Suvorov, 2009; Wagner, 2008, 2010), studentrecorded
video for oral formative assessment is relatively unexplored.
In this study, I examined 15 teachers’ ratings of speech recorded by 39
ESL learners to see if teachers assess speech differently depending on
whether it is presented with visuals. The learners recorded 4 speech
samples: 2 with webcams, 2 with microphones only. A third speech
condition was created by removing the video track from the webcam
recordings, resulting in 3 conditions and 6 samples for each individual.
The teachers rated all 6 samples. I used repeated-measures ANOVAs to
determine whether the teachers assigned significantly different scores
based on the speech conditions. The results showed that the teachers
rated the audio stripped from the video significantly higher than the
video/audio recordings (p = .004, d = .38). This suggests that teachers
may be biased in favor of audio-only recordings and that teachers
should not give students an option of making either an audio or video
recording for a given formative assessment. Further analyses examined
how the students’ and teachers’ preferences for audio-only or video
recordings were related to the ratings.