Audio-synchronized textual enhancement in foreign language pronunciation learning from videos

Abstract

The benefits of multimodal input on foreign language listening comprehension and vocabulary learning are well-established, but only recently has its impact on pronunciation been explored. In this study, we audio-synchronized the highlighting of target words in captions to promote the activation of their phonolexical representations at the time they are auditorily processed and improve phonological updating in the mental lexicon. We recorded the eye movements of 58 L1-Spanish/Catalan learners of English as they watched two videos with target words (TWs) highlighted 500 ms or 300 ms before auditory onset, highlighted from caption onset or, alternatively, under one of two control conditions (unenhanced and uncaptioned). We assessed updating of phonolexical forms in terms of more accurate and faster rejection of mispronunciations of the TWs from pre-to post-test. Results showed that 300 ms synchronized enhancement and unsynchronized enhancement led to longer fixation duration, unsynchronized enhancement reduced TW skipping probability, and both synchronized conditions promoted higher audio-visual synchrony in learners' caption reading. While only the unsynchronized condition resulted in more accurate responses at post-test, all enhancement conditions led to significantly faster rejection of mispronunciations. These initial findings call for further research on audio-synchronized enhancement and its potential benefits for L2 pronunciation learning.