Sparkles on Finger Tips
Skills: Unity, OpenCV, Mediapipe, ML.NET, Python, C#
This project is a prototype of a 2D platform video game. In the game, I used OpenCV, Mediapipe and ML.NET to create the function of hand/finger recognition and symbol recognition.
The idea behind this game is to let the player really feel that he/she is the one drawing and casting the spells that the avatar is drawing and casting. Therefore, the player was allowed the ability of casting spells by making gestures with his/her hand instead of the mouse. At first, my intention was to allow the player to casting spells with a combination of symbols. For example, if symbol A represents big and symbol B represents fire ball, then the player can combine A and B to create a big fire ball. However, in order for the program to distinguish that many symbols, a lot of training data has to be fed to the training model. However, since I made my own training data, it would take me too much time to do so, so I changed the plan and made each symbol represent a spell.
The idea of the protagonist came from Douxie, who is a character in a TV show called Tales of Arcadia. The protagonist of the game is a witch who has a cool device on her wrist that can contain spells she draws. When needed, she can swipe the interface of the device to choose a spell she wants to use and then cast it.
In the game, using their index finger to draw in front of the webcam, the player can draw spells, which will be stored and then used to attack enemies later.
Then, the player can choose spells by swiping both index and middle fingers in the air.
Finally, the player can cast the spell and attack the enemies by holding and then releasing their hand.
In the project, in order to achieve the mechanic in my mind, the function of detecting the player's hands and fingers had to be achieved in Unity. It was fulfilled by using Google's mediapipe, which is a very useful but not yet completed API made with OpenCV that can be used for visual recognition. However, the problem is that mediapipe doesn't support C# and Unity, so I had to use mediapipe in Python and create a .pyd file and then embedded a whole Python environment into my Unity project in order to use the .pyd file in my project. After that, ML.MET and the training data that I made to teach Unity were used to classify the symbols that the player draws.
Overall, the outcome of this project in satisfying. I've achieved the mechanic in my mind, but I have to admit that the function of the hand recognition is not smooth enough and the prototype can be polished a little more. Except that, I think the notion itself is great and the building process of the prototype is rewarding.