Kinect piano in action at Oslo Maker Faire (2014, not 2013)
About the Kinect Piano
The Kinect is a depth sensor camera, which is normally used with the Xbox game console. It can also be connected to a regular computer, and people can make their own applications for it. My projects were inspired by two main ideas:
1. Chris Vik's musical Kinect rig:
Using his Kinect, Chris uses his movements to trigger a digitally controllable 4-storey church organ.
Using his Kinect, Chris uses his movements to trigger a digitally controllable 4-storey church organ.
2. Soundbeam:
The Soundbeam is an infrared sensor which can be connected to digital musical instruments.
The device is used in musical therapy, and can even detect slight tilts of the head. I had the opportunity to see it used by someone who was paralyzed from the neck down, jamming with a music therapist who played the guitar.
The device is used in musical therapy, and can even detect slight tilts of the head. I had the opportunity to see it used by someone who was paralyzed from the neck down, jamming with a music therapist who played the guitar.
1. Exhibition project
I was fortunate enough to have my entry accepted for Maker Faire Oslo 2014. I made it with the visual programming environment vvvv, and controlled it with a simple TouchOSC interface on my tablet.
Demonstration of the setup's capabilities. There's also a seated mode, which makes it usable from a wheelchair.
A number of people tried to "play" individual notes, which is understandable, given the "piano" part of the project's name. This was originally intended to be a feature, but implementing a reliable trigger mechanism proved tricky, even when only tailored to my movements.
However, with some quick instructions, people seemed to really enjoy dancing around and experiencing their movements as the cause of the sound.
2. Prototype I - speech recognition
The first prototype already had the final idea in place: traversing a virtual piano keyboard, with harmonising notes only. I severely overestimated my programming skills and patience, and ended up using NI Mate instead.
I had seen examples of Kinect speech recognition before, but it was easier to just use a regular microphone with Windows' speech recognition solution. Speech recognition is a pain in the ass, and I needed a few recording attempts before it understood all my commands. I could have gone with Jarvis, but I quite like Lil Jon.
- http://www.ni-mate.com/ (Functionality)
- http://www.nerds.de/en/download.html (MIDI from NI Mate to Ableton Live/FL Studio/Logic/whatever)
- https://www.microsoft.com/en-us/download/details.aspx?id=13045 (Speech recognition add-on)
3. Prototype II - gestures
Gestures are like the staple of science-fiction interfaces. Of course I wanted to have a go at them! FAAST was perfect for the task.
Now that I'm writing this, I don't know why I didn't include the "hip thrust -> lazer sound" gesture I set up.
4. Final thoughts on the Kinect
These projects were challenging to set up. I was able to successfully calibrate them to my own movements, but this didn't necessarily transfer to other people's movement patterns.
Things to be improved
Even if I could implement some kind of adaptive system based on people's proportions, the main problem lies in the gap between what users think, and what the hard numbers say. If I ask someone to repeat what I just did, or even to repeat the same movement 10 times, the numbers would reveal that they're significantly different. In other words, the setup wouldn't act in accordance with people's expectations.
The Kinect used as a precise controller is kind of unreliable. Even though nearly all of my personal interactions are successful, just one or two misses feel unnecessary, and wouldn't happen by pressing a visible button. At times, it felt like turning on the light by clapping instead of just pressing a light switch.
Another really frustrating problem I had was false positives - when the camera picks a chair or something and thinks it's a person - assigning it the role of player 1. These mostly happen in the background, and can be avoided by only receiving coordinates if the player's Z coordinates are less than X meters away, but then users aren't allowed to move as freely.
Things that were really cool
I got to see so many different people play music with their movements! I saw little kids jumping and dancing with it, taking turns being dancer and conductor, and my grandma had one of the most energetic and diverse performances all day at Maker Faire.
My master's project employed an alternative version with either calm or loud sounds. I got to have "pianos and explosions" in my title, and I had so much fun testing it out on people. It was definitely much more fun than reading by myself in the library. Surprisingly, my tests didn't reveal any difference in activity levels, but I'd love to have those findings examined further.
Anything can control anything. If I can set up a gesture to trigger and elaborate sequence of keyboard shortcuts, sky's the limit.
Next project
Recently, I've had a go at randomly positioned piano keys, which change position each time a note is triggered, to make it more interesting. This new project includes a "magnet" feature, which gently pulls the hand-controlled point towards its nearest virtual piano key, as it starts getting closer.
This feature could mitigate some lack of physical ability or precision, which have previously shown to be challenging to deal with. However, the "magnet" can quickly become too strong - thus equally annoying as the excessive amount of precision often required by the Kinect.
For music projects, I believe the Kinect is better suited to control continuous signals - basically anything that is normally controlled with a knob or slider. Switch signals can also work, but I've found these to be tricky to set up - how do you decide what's on and off?
The best solution would probably be to have a musical foundation already in place, with the sensor dealing with post-processing or "non-critical" musical elements.
Please take my ideas and make something cool!
/ Kristian
Please take my ideas and make something cool!
/ Kristian
