Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

• DOI: 10.48550/arxiv.2309.15223
• Publication Date: 2023-01-01
• Authors: Yu Yu; Chao-Han Huck Yang; Jari Kolehmainen; Prashanth Gurunat...; Yile Gu; Sungho Ryu; Roger Ren; Qi Luo; Aditya Gourav; I‐Ming Chen; Yi-Chieh Liu; Tuan Dinh; Ankur Gandhe; Denis Filimonov; Shalini Ghosh; Andreas Stolcke; Ariya Rastow; Ivan Bulyko
• Keywords: Discriminative model; Regularization; Rank (graph theory); Domain Adaptation

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up pretraining stage and adapting to specific domains limit their practical use Here we present method decomposition train rescoring model adapt it new using only fraction (0.08%) parameters. These inserted matrices are optimized through discriminative training objective along with correlation-based regularization loss. The proposed Rescore-BERT (LoRB) architecture is evaluated LibriSpeech internal datasets decreased times by factors between 5.4 3.6.