Pushing the limits of remote RF sensing by reading lips under the face mask

• DOI: 10.1038/s41467-022-32231-1
• Publication Date: 2022-09-08T12:10:37Z
• Authors: Hira Hameed; Muhammad Usman; Ahsen Tahir; Amir Hussain; Hasan Abbas; Tie Jun Cui; Muhammad Ali Imran; Qammer H. Abbasi
• Keywords: Face masks

The problem of Lip-reading has become an important research challenge in recent years. goal is to recognise speech from lip movements. Most the technologies developed so far are camera-based, which require video recording target. However, these have well-known limitations occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based not useful for multi-modal hearing aids coronavirus (COVID-19) environment, where face masks a norm. This paper aims solve fundamental camera-based systems by proposing radio frequency (RF) based framework, having ability read lips under masks. framework employs Wi-Fi radar as enablers RF sensing Lip-reading. A dataset comprising vowels A, E, I, O, U empty (static/closed lips) collected using both technologies, mask. data used train machine learning (ML) deep (DL) models. high classification accuracy 95% achieved on utilising neural network (NN) Moreover, similar VGG16 model radar-based dataset.