Knowing when games have made a change
The Harmony Institute researches the influence of entertainment. Our organization is founded on the premise that stories are powerful tools for creating social change. We’re interested in understanding how films, novels, television shows, and games affect their audiences and finding the most valuable metrics for each medium. It is these metrics, more than anecdote or assumption, that provide clear and accurate answers to the fundamental questions of what storytellers hope to achieve when they create entertainment.
Movies, books, and television shows were, until recently, how most stories were told to mass audiences. Those mediums all move in one direction--they deliver a complete story to an audience. Gaming, on the other hand, is participatory; it’s a two-way street in which players have agency, or control, in the story. This participation can contribute to audiences’ increased learning or retention of information about the narrative. It’s this participatory aspect, and its ramifications for interactive data analysis that we’re most excited about.
In the study of games, there is a tendency to treat them just like movies or novels and focus only on the end results, such as how many people played the game or whether the overall experience was positive. However, the interactive nature of games offers many more benchmarks to assess influence. Every level completed or player action can contribute to the measurement of that game’s effect on audiences. This is different than playability testing, where game testers report on whether the game was an enjoyable experience or how well the mechanics of gameplay worked.
To explain the evaluation of entertainment influence in more concrete terms, we’ve chosen three game examples that are each engaging in different ways. We’ll walk through the particular ways in which they seek to create change in their audience, as well as give examples of how we might measure those changes.
Movies, books, and television shows were, until recently, how most stories were told to mass audiences. Those mediums all move in one direction--they deliver a complete story to an audience. Gaming, on the other hand, is participatory; it’s a two-way street in which players have agency, or control, in the story. This participation can contribute to audiences’ increased learning or retention of information about the narrative. It’s this participatory aspect, and its ramifications for interactive data analysis that we’re most excited about.
In the study of games, there is a tendency to treat them just like movies or novels and focus only on the end results, such as how many people played the game or whether the overall experience was positive. However, the interactive nature of games offers many more benchmarks to assess influence. Every level completed or player action can contribute to the measurement of that game’s effect on audiences. This is different than playability testing, where game testers report on whether the game was an enjoyable experience or how well the mechanics of gameplay worked.
To explain the evaluation of entertainment influence in more concrete terms, we’ve chosen three game examples that are each engaging in different ways. We’ll walk through the particular ways in which they seek to create change in their audience, as well as give examples of how we might measure those changes.
Did the player learn something?
One of the ways we assess impact is if the audience demonstrated a change in what they know about the subject being presented. What makes games unique as learning vehicles? In the gaming world, this question is often the focus of the “serious games” genre, which applies interactive design principles to explicit educational or social issues. However, almost any game can engage these questions. To further explore how games can approach such issues in novel ways, we looked at Dys4ia, an 8-bit style indie game that explores the difficult details of undergoing hormone-replacement therapy and transitioning from one gender to another. Players are exposed to multiple mini-games along a narrative filled with emotional cues and medical facts.
The game creator, Anna Anthropy, adeptly describes the game play: “This was a story about frustration - in what other form do people complain as much about being frustrated? A video game lets you set up goals for the player and make her fail to achieve them. A reader can’t fail a book. It’s an entirely different level of empathy.”
One of the ways we assess impact is if the audience demonstrated a change in what they know about the subject being presented. What makes games unique as learning vehicles? In the gaming world, this question is often the focus of the “serious games” genre, which applies interactive design principles to explicit educational or social issues. However, almost any game can engage these questions. To further explore how games can approach such issues in novel ways, we looked at Dys4ia, an 8-bit style indie game that explores the difficult details of undergoing hormone-replacement therapy and transitioning from one gender to another. Players are exposed to multiple mini-games along a narrative filled with emotional cues and medical facts.
The game creator, Anna Anthropy, adeptly describes the game play: “This was a story about frustration - in what other form do people complain as much about being frustrated? A video game lets you set up goals for the player and make her fail to achieve them. A reader can’t fail a book. It’s an entirely different level of empathy.”
Dys4ia is a great example of how to use games to increase an audience’s comprehension of an issue. The stylized abstraction made a difficult issue approachable, and stuttering game mechanics contributed to the sense of player frustration that Anthropy discussed.
Context is one of the great challenges of social science research. It’s hard to measure social behavior in the confines of a lab or through a simple survey. However, games like Dys4ia provide a natural setting for understanding how players respond to the social messages embedded in games. How might we imagine adapting gameplay so that we could measure an audience’s understanding of the issues that transgender people face? A developer might be able to further the experience with an additional mini-game at the end of the formal game. This mini-game could present choices to the user that reveal his or her knowledge of past lessons. Another option might be to pinpoint a current section of the game and examine how users respond to the tasks by measuring click counts and cursor movement. By staying within the context of the game and using easily measurable data, we can embed evaluation directly into play.
Did the player feel something?
Dys4ia teaches the player about the intricacies of hormone-replacement therapy, but it has the capacity to convey the emotional challenges of that process. As Anna Anthropy says, frustration is a key feeling for audiences of her game. Likewise, gauging an audience’s attitude or emotional understanding of the subject is an important metric of impact. Games with a focus on shifting emotions may have an immersive storyline with strong simulation qualities, a visual aesthetic that mirrors the desired emotions, and a focus on character development. One game released in 2012 was particularly illustrative of these qualities: Spec Ops: The Line. The game is a triple-A title with high-quality action and an immersive gameplay environment. The storyline initially leads players through a seemingly standard war game, based in Dubai, and progressively introduces emotionally complex morality choices. Imagine experiencing firsthand the plot of Heart of Darkness or Apocalypse Now. While not explicitly about a social issue, the game has started a dialogue, with many reviewers agreeing that the game offers a commentary on war, or more specifically, the way audiences approach war video games.
From an evaluation perspective, Spec Ops has a highly affective quality and offers researchers the potential to measure player changes in attitudes about shooting, violence, and war. For example, an evaluation could include monitoring comments in game forums, or even player-created Spec Ops YouTube videos to determine which scenes players were discussing and whether these scenes drew on players’ skills in the game mechanics or their decisions in how to advance the narrative.
Another study might track individual participants as they progress through the game and note their actions at pivotal game benchmarks, such as whether the number of casualties decreases substantially over time, or whether highly-skilled players begin to waver from ‘good’ gameplay as the story becomes increasingly more disruptive. For example, do the players stand still in the game for longer stretches or does their movement become more erratic? Are they more likely to forget game commands or press buttons accidentally as the story line changes? All of these metrics could be used in a study assessing the effects of the game’s morality on a player’s cognitive load, or the number of interactions his or her brain and working memory can process at one time.
Did the player do something?
A third metric we use in impact assessment is measuring whether an audience feels compelled to actively interact with the material, like donating to a documentary’s awareness campaign. For traditional forms of media, we generally assess audience responses after they’ve seen a movie or read a book. For games, we can also look as the actions they take during gameplay, such as spending money or points on character upgrades, or the number of specific types of clicks. To that extent, casual games have exploded in popularity and prove that simple, addictive gameplay is a powerful mechanic for enticing players to take action in the real world. Take for example, the game Food Force.
HI researchers can (anecdotally) attest this combination is addictive. We also found the relatively large scope of the game led to interesting discoveries and real-world learning opportunities, which kept us playing. Most importantly, players can purchase items to advance their game. Food Force then donates that money to fight world hunger. Players can track how many children they’ve helped with the “real world impact tracker”.
As a highly interactive game, Food Force lends itself to quantitative studies, such as tracking the time participants spend in the game, the number of sessions they play, the frequency of return sessions, the number of items acquired, the number of trades players make with friends, and the number of times the game is shared online. These metrics can then be correlated against items like the amount of money players raise in game, how widely they recruit other players on Facebook, or a player’s likes and interests.
Designing with evaluation principles
Despite a developer’s best intentions, a game cannot be all things to all people. This is why it’s important to design with specific audiences and goals in mind. Taking this a step further, designers can also think about how they might iterate and improve their games based on the feedback from impact evaluation metrics, just as they do with the beta-testing process. Even if a game developer decides not to undertake a formal assessment, it is useful to think of evaluation goals as a guiding development tool. For example, when designing a game to have a message, think about whether the factual information you’re trying to convey adds to the narrative or distracts from the player experience. Can you can simplify that information to create a more immersive story? Ultimately, the identification of these goals may even help to clarify the purpose of a game and what it can evoke in its audience.
At their best, games are playful and interactive. They can realistically or fantastically simulate situations and events. They encourage strategic thinking and problem solving. For these reasons and more, games have the potential to act as an important force in our society. Accordingly, they demand rigorous, research-based impact analysis that reflects their unique properties and widespread appeal.
Images: jayisgames.com | diygamer.com | gamechurch.com | blog.games.com
