An ensemble learning-enhanced multitask learning method for continuous affect recognition from facial images

• DOI: 10.1016/j.eswa.2023.121290
• Publication Date: 2023-08-31T07:15:00Z
• Authors: Ercheng Pei; Zhanxuan Hu; Lang He; Hailong Ning; Abel Díaz Berenguer
• Keywords: Affect; Leverage (statistics); Ensemble Learning; Affective Computing; Transfer of learning

Continuous affect recognition from facial images aims to estimate the values of multiple affective dimensions from a facial image sequence. To leverage relevant information between multiple affective dimensions, multitask learning has been used in the estimation of continuous affective states. Most of the existing multitask continuous affect recognition methods focus on designing elaborate multitask networks. Meanwhile, a few research works consider using multitask training strategies for continuous affect recognition. In general, existing multitask continuous affect recognition methods face the problem of unstable training effects. In this work, to improve the stability of multitask learning, we propose an ensemble learning-enhanced multitask network architecture for continuous affect recognition. In addition, we introduce a novel adaptive weighted loss-based multitask learning strategy to effectively train the proposed multitask continuous affect recognition model. Experimental results, on the RECOLA, SEMAINE and AFEW-VA datasets for continuous affect recognition, demonstrate the potential of the proposed method compared to state-of-the-art methods.