Immersive application for real-time interactive music performances using spatial audio

This paper proposes the implementation of an Extended Reality (XR) application that enables real-time streaming of live music concerts, where users equipped with XR headsets can interact with musicians in immersive virtual spaces. Musicians are represented by avatars, each associated with one or more audio tracks captured by a digital mixer. Using spatial audio and real-time interaction mechanisms, the system provides an engaging and dynamic concert experience, simulating the presence of a live audience while offering an user-controlled feedback channel. In the proposed system, multitrack audio streams from the musicians’ mixer are transmitted to the XR environment via low-latency streaming protocols. These audio tracks are spatialized in the 3D space using advanced algorithms such as Head-Related Transfer Function (HRTF)-based rendering or Ambisonics within typical 3D game engines (e.g. Unity, Unreal). Each musician’s avatar, positioned in the virtual environment, dynamically updates its location and sound directionality relative to the musician’s position and orientation. Audience members can interact with musicians in real time by providing audio feedback (e.g. cheering, clapping, singalong), which are reflected on the musician’s side by spatialized audio crowd responses.