Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models

• DOI: 10.48550/arxiv.2502.10378
• Publication Date: 2025-01-01
• Authors: Ding, Jiexin; Zhao, Bowen; Wang, Yuntao; Liu, Xinyun; Hao, Rui; Chatterjee, Ishan; Shi, Yuanchun
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction; Computation and Language (cs.CL); Human-Computer Interaction (cs.HC)

English as a Second Language (ESL) learners often encounter unknown words that hinder their text comprehension. Automatically detecting these words as users read can enable computing systems to provide just-in-time definitions, synonyms, or contextual explanations, thereby helping users learn vocabulary in a natural and seamless manner. This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%. We implemented a real-time reading assistance prototype to show the effectiveness of EyeLingo. The user study shows improvement in willingness to use and usefulness compared to baseline methods.