Transcribing Natural Languages for the Deaf via Neural Editing Programs

• DOI: 10.1609/aaai.v36i11.21457
• Publication Date: 2022-07-04T10:14:39Z
• Authors: Dongxu Li; Chenchen Xu; Liu Liu; Yiran Zhong; Rong Wang; Lars Petersson; Hongdong Li
• Keywords: Spoken Language; Transcription

This work studies the task of glossification, which aim is to em transcribe natural spoken language sentences for Deaf (hard-of-hearing) community ordered sign glosses. Previous sequence-to-sequence models trained with paired sentence-gloss data often fail capture rich connections between two distinct languages, leading unsatisfactory transcriptions. We observe that despite different grammars, glosses effectively simplify ease deaf communication, while sharing a large portion vocabulary sentences. has motivated us implement glossification by executing collection editing actions, e.g. word addition, deletion, and copying, called programs, on their counterparts. Specifically, we design new neural agent learns synthesize execute conditioned sentence contexts partial results. The imitate minimal exploring more widely program space via policy gradients optimize sequence-wise transcription quality. Results show our approach outperforms previous margin, improving BLEU-4 score from 16.45 18.89 RWTH-PHOENIX-WEATHER-2014T 18.38 21.30 CSL-Daily.