Facial and Speech-based Signal Processing Systems for Quality of Experience and Emotion Estimation of Multimedia Applications

Assessing the Quality of Experience (QoE) of multimedia services is crucial to ensure end users are satisfied. However, traditional feedback collection may suffer from bias due to the rating scale and may neglect the dynamic subjective nature of human perception, influenced by the user’s emotional state. This paper proposes an alternative solution to unobtrusively estimate QoE relying on features related to the user’s facial expressions and speech characteristics, which naturally reflect the user’s emotional state and enable QoE estimation without explicit feedback. The presented solution includes several steps, from the data collection process to the extraction and selection of the facial and speech features used to train the resulting QoE estimation models. Both single-modal and multi-modal learning approaches based on data fusion have been considered. These models can support the management of network and application resources of traditional and immersive multimedia services.