Enhancing XR theatre with remote scent delivery using audio speech recognition and convolutional neural network-based scene detection

This paper presents a multi-sensory solution designed to enhance the immersive experience of remote audiences for theatrical plays. Using the Macbettu play as a case study, the approach synchronises olfactory and visual stimuli for remote viewers. The solution considers two approaches for generating scents: audio detection that identify the scenes within the play that require specific scents, based on automatic speech recognition (ASR), and convolutional neural networks (CNN) for scene recognition triggering the scent dispensers. When the key scenes are detected during the live play, via network communications, olfaction dispensers release the corresponding scent for remote users viewing the play on VR headsets. Real-time communication is managed through an MQTT-based approach. The paper also compares user feedback for CNN-based recognition versus the audio-based approach, focussing on scent delay and user perceived Quality of Experience (QoE) of the generated remote scents in multiple distances from a scent source. This work illustrates how scent-enhanced XR technologies can be integrated into virtual theatre experiences to engage remote audiences.