Kusmos Live: Ambient stream

•

Reportedly, artists only benefit from quarantines and isolation, especially if they are ready to change themselves and to look for new possibilities. In spring 2020 our Kuflex studio began an experimental project Kusmos Live.

The purpose of our experiment is to upgrade our Kusmos system in order to create a vivid visual content of online home concerts and to turn them into an interactive digital show.

What is Kusmos? The original Kuflex program was designed for music concerts.

In 2018-2019 we used this technology in some common projects together with Sila Sveta studio on the Therr Maitz concert, Caprices 2018 techno festival, Nina Kraviz experimental performance at Coachella 2019. With the Kusmos system we capture the artist (via depth-sensors and a webcam) and transfer the resulting 3D-model into the digital scene, adding visual effects in real time. In the current situation when all events and music festivals are being cancelled around the world we decided to upgrade the program with some new effects and control functions online. This is how Kusmos Live project has started.

3D-scene created in Unreal Engine

What is Kusmos Live? In April and May 2020 Kuflex set up two online concerts together with a musician called Leksha. The concerts were video streamed on YouTube and VK. Between the concerts our team has completed the Kusmos program with some new features and plans to keep on doing so.

How does it work? The Tracker program receives depth-sensor data, calibrates a point cloud as we need it (we can rotate the cloud, cut off everything you don't need keeping only the data of the artist oneself and merge point clouds from two sensors) and sends it to the renderer (UE4 scene). Scene functions build a 3D model using the data provided by the Tracker and then we can layer all kinds of features with effects on the model and also transform and distort it according to the artistic concept. We also captured video from the laptop's webcam and sometimes showed its picture. 3D scene acting as the artist surrounding, a set of virtual cameras for capturing from different viewpoints and visual effects were set up in advance in the Unreal Engine. Then the OBS program captures the video of the launched scene and sends it to the video streaming server.

3D-scene created in Unreal Engine. Human model created by using the data from the Tracker

How does interactivity work? During the second concert we tested some new features for the viewers interactive communication with the stream. While Leksha (Smolensk, Russia) was playing his ambient-set, a VJ (Moscow, Russia) was controlling visual effects with the use of commands via YouTube chat in real-time mode. Our team has implemented this function between the concerts and decided not to tell the viewers about it. During the stream, noticing weird messages in the chat, some of the viewers started to realize that they could not only send a message to the chat but to even affect visualization. In the end the concert has turned into a digital quest. Some viewers picked up effect control by sending particular commands to the chat. We have yet to comprehend how to develop this function in the future.

Blueprint scene program in Unreal Engine

What is required for the technical implementation? For the first online concerts we used:

- Kinect for Xbox One (one or two sensors to create a model of an artist),

- gaming PC or laptop,

- music setup of the artist,

- OBS software (free version for live streaming and video recording),

- Unreal Engine 4 to create and to render a 3D-scene with visual effects (free gaming engine from the Epic Games),

- additional functions and plugins for Unreal Engine (Kuflex proprietary design),

- Tracker program for receiving a 3D model of the artist realtime (Kuflex proprietary design).

Tracker program window with a point cloud of a man scanned by the Kinect sensor

What’s next? Our team plans to make new Kusmos Live concerts and to make adjustments to the program that can later become a universal tool. Using it can turn any concert - online or offline, into a magical interactive show where the viewers can participate by altering content in real-time. Follow the announcements of our experiment.

And last but not least, we kindly ask you to leave your feedback by answering a few questions. This will help us to make Kusmos a lot more interesting for you.

Keeping in touch! See you live!

KU!

Kuflex team