Output-associative RVM regression for dimensional and continuous emotion prediction

• DOI: 10.1016/j.imavis.2011.12.005
• Publication Date: 2012-01-10T14:51:20Z
• Authors: Mihalis A. Nicolaou; Hatice Gunes; Maja Pantic
• Keywords: Dimensional and continuous emotion prediction; Output-associative RVM regression; Gaussian distribution; HMI-MI: MULTIMODAL INTERACTIONS; IR-79503; 02 engineering and technology; EC Grant Agreement nr.: FP7/211486; EWI-21348; Facial expressions; METIS-285040; Kernel; EWI-22939; Audio cues; Shoulder movements; Support Vector Machines; 0202 electrical engineering, electronic engineering, information engineering; Feature extraction; Training; METIS-296242; Noise; IR-84217; Estimation

Many problems in machine learning and computer vision consist of predicting multi-dimensional output vectors given a specific set of input features. In many of these problems, there exist inherent temporal and spatial dependencies between the output vectors, as well as repeating output patterns and input–output associations, that can provide more robust and accurate predictors when modeled properly. With this intrinsic motivation, we propose a novel Output-Associative Relevance Vector Machine (OA-RVM) regression framework that augments the traditional RVM regression by being able to learn non-linear input and output dependencies. Instead of depending solely on the input patterns, OA-RVM models output covariances within a predefined temporal window, thus capturing past, current and future context. As a result, output patterns manifested in the training data are captured within a formal probabilistic framework, and subsequently used during inference. As a proof of concept, we target the highly challenging problem of dimensional and continuous prediction of emotions, and evaluate the proposed framework by focusing on the case of multiple nonverbal cues, namely facial expressions, shoulder movements and audio cues. We demonstrate the advantages of the proposed OA-RVM regression by performing subject-independent evaluation using the SAL database that constitutes naturalistic conversational interactions. The experimental results show that OA-RVM regression outperforms the traditional RVM and SVM regression approaches in terms of accuracy of the prediction (evaluated using the Root Mean Squared Error) and structure of the prediction (evaluated using the correlation coefficient), generating more accurate and robust prediction models.