Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach

• DOI: 10.48550/arxiv.2401.10747
• Publication Date: 2023-12-28
• Authors: Liu, Weide; Zhan, Huijing; Chen, Hao; Lv, Fengmao
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Sound (cs.SD); Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Audio and Speech Processing (eess.AS); FOS: Electrical engineering, electronic engineering, information engineering; Computation and Language (cs.CL); Computer Science - Sound; Electrical Engineering and Systems Science - Audio and Speech Processing; Machine Learning (cs.LG)

Multimodal sentiment analysis aims to identify the emotions expressed by individuals through visual, language, and acoustic cues. However, most of the existing research efforts assume that all modalities are available during both training and testing, making their algorithms susceptible to the missing modality scenario. In this paper, we propose a novel knowledge-transfer network to translate between different modalities to reconstruct the missing audio modalities. Moreover, we develop a cross-modality attention mechanism to retain the maximal information of the reconstructed and observed modalities for sentiment prediction. Extensive experiments on three publicly available datasets demonstrate significant improvements over baselines and achieve comparable results to the previous methods with complete multi-modality supervision.<br/>We request to withdraw our paper from the archive due to significant errors identified in the analysis and conclusions. Upon further review, we realized that these errors undermine the validity of our findings. We plan to conduct additional research to correct these issues and resubmit a revised version in the future<br/>