Biased Self-supervised Learning for ASR

• DOI: 10.21437/interspeech.2023-2499
• Publication Date: 2023-08-14T04:22:20Z
• Authors: Florian L. Kreyssig; Yangyang Shi; Jinxi Guo; Leda Sari; Abdel-rahman Mohamed; Philip C. Woodland
• Keywords: FOS: Computer and information sciences; Sound (cs.SD); Computer Science - Computation and Language; Computer Science - Artificial Intelligence; Computer Science - Sound; 03 medical and health sciences; Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS); FOS: Electrical engineering, electronic engineering, information engineering; 0305 other medical science; Computation and Language (cs.CL); Electrical Engineering and Systems Science - Audio and Speech Processing

Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.<br/>Submitted to ICASSP 2023<br/>