Hybrid TFT–CPC-Based Architecture for Real-Time Emotion Prediction from Music via Physiological Signal Integration

Music has a great impact on human emotions which are expressed by complex physiological reactions. Accurate real-time emotion prediction from music is still challenging, because of the heterogeneity of multimodal signals as well as the temporal dependency and the inter-individual variability. This work is oriented towards proposing a coherent and interpretable framework to real-time emotion prediction using physiological information and music features fusion. A hybrid architecture consisting of Contrastive Predictive Coding (CPC) for Physiological representation learning (self-supervised learning) and Temporal Fusion Transformer (TFT) for long-range modeling is proposed. Multimodal signals such as EEG, ECG GSR, Respiration and Music Audio signals are temporally synchronized and adaptively preprocessed. The continuous valence- and arousal states are predicted in this model by adaptive attention and online personalization. Experiments performed on the DEAP data set show better performance with valence and arousal MAE of 0.084 and 0.091, respectively, and an overall accuracy rate of 89.6%, which is higher than state-of-the-art methods. The effectiveness of hybrid self-supervised and transformer based fusion in real-time affective computing applications is confirmed by the results.