A design project that aims to change the audience's virtual experience of a physical street dance battle during a pandemic era through discovering opportunities for immersion, interaction, and collaboration within a hybrid integration of communication systems.
This project seeks to uncover what it means to be virtually present in a physical street dance battle and how social connectedness can be fostered within the street dance community even when the environment in which we sense other bodies has been profoundly altered. The communicative capacity of street dance battles are examined through the viewpoint of a sender and receiver, referring to the dance and the audience respectively. By examining the effectiveness of current approaches in digital communication used to facilitate street dance battles today, the ensuing deliverable presented aims to reveal viable opportunities for better experiential design for key participants of street dance battles.
While the main focus of this project is targeted towards the experience of audience members at street dance battles, the relationship between the performer and audience should not be discounted. Street dance creates an opportunity for complex storytelling where bodies within the battle-space send and receive messages at the same time. The communicative capacity of street dance battles are examined through the viewpoint of sender and receiver, referring to the dancer and the audience respectively.
The live streamed street dance battle offers a shared and novel experience for audiences but lacks the interactivity and immersiveness that could trigger stronger feelings of being in the presence of others even while watching alone. The project aims to present audiences with an engaging experience as they may feel disconnected with other participating bodies in the street dance battle due to the lack of immersiveness and interactivity from watching live streamed street dance battles.
Intensive research was conducted to uncover the reciprocative phenomena that happens during street dance battles between bodies. Surveys were conducted amongst street dancers, inquiring how they felt at street dance battles - physical, online and hybrid. Analysis and live observations of street dance battles - physical, online and hybrid were also conducted. One of the more extensive research was done during an interview with two main subjects.
They were placed in the shoes of a performer and an audience. Questions that touched on intentions and emotions were inquired. Further questions were asked that touched on the differences they felt were apparent in an online and a physical street dance battle.
At the end of the interview, an experiment was conducted with the subjects to examine on a small scale, how bodies involved in a street dance battle communicate and react through dance in different environments. The subjects were placed in the shoes of a dancer and the audience for the experiment.
The responses were compiled into a publication which delved deeper into the mechanics behind communicating bodies in a dance battle space, where the self is heavily influenced by its environment, and the environment, made up of the bodies that exist in it. The responses reinforced the significance of active communication between participating bodies, in a culture that heavily rely on kinesthetic empathy.
All videos were exported in 3fps and the dancer’s skeletal frame was traced manually from the video’s exported frame afterwards.
The audience’s physical presence plays a huge part in creating a memorable atmospheric environment in street dance battles, influencing other participating bodies present. A communal sense of unity is cultivated amongst individuals through emotions, behaviour and a shared interest in street dance. Due to the social distancing restrictions emanating from the COVID-19 pandemic, the physical separation of audiences and performers take on more extreme proceedings. Participants of street dance battles feel a disconnect between themselves and others, changing the experiences and extent of interpersonal connections. Audiences rely on live stream broadcasting of street dance battles to feel involved and less socially isolated from the culture. By exploring new possibilities in this recent hybrid street dance battle phenomenon and expanding on the culture’s communicative and interactive nature, audience members could be introduced to a more immersive and interactive experience when watching live street dance battles virtually.
Naturally, a deeper insight to the culture can be inferred through the emotions and related phenomena exchanged using nonverbal communication. Dance is a form of nonverbal communication between the dancer and his own body, and, between the dancer and other people. The dancer sends intended messages through movement and the audience receives it, replying and acknowledging the message by sending signals that gives the dancer reciprocal information. One noticeable feature are gestures, thrown during a battle round by anyone and everyone at the event. These gestures carry meanings that are exclusive to street dance culture.
The expansion of how the audience performing gestures could form a sort of nonverbal reply and give dancers a sense of directive was as inspiration to the following deliverable. Through a series of primary research, it was observed that a majority of participants who were questioned on whether they felt engaged when watching a live street dance battle through a screen felt personally disengaged from the whole experience. The majority of the survey responses also expressed the lack of immersiveness and interactivity from watching live streamed street dance battles. Currently, the only way audiences can participate in the event at higher degree is through engaging in the live chat function on live streams. Though, this form of engagement does not reach the dancers even though the contents of the reactions are mostly directed towards them.
Could we create something that instantly responded to an audience’s reaction moreover informing the dancers in real time? The idea of creating a channel that facilitates live feedback between the dancer and audience was established through the deliverable.
A street dance gesture recognition model was trained through machine learning to recognise three gestures in particular; props, point and thumbs up. The process started with utilising an open-sourced three-dimensional hand pose detection model from the TensorFlow.js library. It detects twenty one different key points on the hand and tracks these key points according to how the hand moves. Because it is a three-dimensional hand pose detection model, the detection is successful no matter the orientation of the hand.
The next step was to teach the model to recognise specific gestures through a neural network. These gestures are commonly seen performed by the audience as a form of positive response towards a dancer’s round. Data in the form of all the key point coordinates of each gesture was used for the training. Each gesture was trained with a hundred epochs. Using algorithms, the neural network can recognize hidden patterns and correlations in raw data, cluster and classify it, and over time, continuously learn and improve.
The trained model relies on a confidence score to classify each gesture when it is done towards a web camera. After many attempts in trying to make the model as accurate as it could be, a successful gesture classification model was created to recognise those three mentioned poses.
Try the model here.
Through deliberation, a visual output was decided to accommodate the data input generated from the three gestures. Since music is integral to a dance battle, having sound as a sensory trigger may interfere with the music which dancers heavily rely on to carry their rounds. Light was settled on for the sensory trigger for dancers. The three gestures are treated as data input and are translated into a live light projection that transforms at the physical street dance battle event with each gesture that is recognised. The projection was intended to be a rather abstract one as having elaborate infographics or expressions would require a considerable amount of attention and focus from the dancer for it to be deciphered. This was something a dancer could not do when they are engaged in a battle, especially when they are the ones dancing.
The projection would happen in real time and the audience performing the gesture from whichever location could see it conveyed through the live stream as it appears during the battle. The performers who are physically present at the dance battle space are able to observe the light projection as it happens. This mirrors the instant auditory feedback from audiences during street dance battles before the pandemic, when a performer does a set of movements that stirs the crowd to cheer verbally. This exchange is reciprocal, with a possibility for the performer to feel more motivated by resounding encouragement from the audience. The choice to work with light projections takes into consideration the ease of application and logistical planning from the standpoint of a street dance organiser. The spaces where street dance battles are usually held include fixed equipment such as stage lighting and projectors. In cases where there is no such equipment that comes with the space, it is not difficult to attain a relatively priced projector.
The light projection changes every time the audience performs one or many of the gestures, intensifying as the number of gestures recognised increases. The gestures were individually set to control a specific part of how the light projection warped and the RGB values of the warped curve.
A video that briefly explains the project was created so as to provide a more visual and kinetic mode of demonstration.
