Creating an Immersive Virtual Orchestra Conducting Experience

Mert Mermerci, Hedvig Kjellström; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 8251-8255

The role of a musical conductor is to coordinate and provide an interpretation to the music performance in an orchestra. Conducting a symphony orchestra is reserved for very few individuals. In the project described here we would like to give more people this extraordinary experience. We are in the process of creating a virtual immersive conducting experience in the visualization dome Wisdome Stockholm in Tekniska the Swedish National Museum of Science and Technology. In the installation the user (the museum visitor) will stand wearing motion capture devices in the middle of the dome theater thus surrounded by the 180? projection of a recording of a symphony orchestra. The recording (made in February this year) depicts the Swedish Radio Symphony Orchestra performing the start of Beethoven's fifth symphony. The captured user motion will be fed into a gesture recognition module which will regress a time signal indicating progress in the recording. The time signal will be fed to a video playback module that plays the recording at the pace controlled by the user motion. The gesture module will be trained with a dataset of motion from different conductors. The system will also feature some gamification in the form of orchestra reactions to success or failure. The installation will be on show in the museum in the beginning of 2025.