Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering

• DOI: 10.21437/interspeech.2023-847
• Publication Date: 2023-08-14T04:22:20Z
• Authors: Heng-Jui Chang; Alexander H. Liu; James Glass
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Audio and Speech Processing (eess.AS); FOS: Electrical engineering, electronic engineering, information engineering; Computation and Language (cs.CL); Electrical Engineering and Systems Science - Audio and Speech Processing

Accepted to Interspeech 2023<br/>Self-supervised speech representation models have succeeded in various tasks, but improving them for content-related problems using unlabeled data is challenging. We propose speaker-invariant clustering (Spin), a novel self-supervised learning method that clusters speech representations and performs swapped prediction between the original and speaker-perturbed utterances. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU. Spin improves pre-trained networks and outperforms prior methods in speech recognition and acoustic unit discovery.<br/>