SimulTron: On-Device Simultaneous Speech to Speech Translation

• DOI: 10.48550/arxiv.2406.02133
• Publication Date: 2024-06-04
• Authors: Alex Agranovich; Eliya Nachmani; Oleg Rybakov‎; Yifan Ding; Jia Ye; Nadav Bar; Heiga Zen; Michelle Tadmor R...
• Keywords: Speech translation

Simultaneous speech-to-speech translation (S2ST) holds the promise of breaking down communication barriers and enabling fluid conversations across languages. However, achieving accurate, real-time through mobile devices remains a major challenge. We introduce SimulTron, novel S2ST architecture designed to tackle this task. SimulTron is lightweight direct model that uses strengths Translatotron framework while incorporating key modifications for streaming operation, an adjustable fixed delay. Our experiments show surpasses 2 in offline evaluations. Furthermore, evaluations reveal improves upon performance achieved by 1. Additionally, achieves superior BLEU scores latency compared previous method on MuST-C dataset. Significantly, we have successfully deployed Pixel 7 Pro device, its potential simultaneous on-device.