For a brief look at the justification and theory behind the project, check out my Material Listening Manifesto (PDF, 13 pages) in which I argue that design can provide the structure for a digital experience with recorded music supported by the embodied presence of the listener and the materiality of the artifact. If you are interested in more research and precedent for the project, as well as in-depth descriptions of my studies, the full document is also available (96 pages, PDF: 1.9 MB).
Still from final study (see animation at bottom of page)
My graduate thesis investigates the nature of our experiences with cultural artifacts in the digital environment. I regard the interface as a cultural form, and assess its relationship to content in our encounters with digital music.
To speak of the interface as "cultural form" is to acknowledge its status as a signifier of human value in a (digital) world where content is largely immaterial and abstract. I believe current systems designed to manage music collections and represent digital packages do not support the kinds of connotative associations that did past media, and I argue that these associations are the precondition for the assignment of value. Without material support, music's cultural import is diminishing. It does not have to be so: this state of affairs, I argue, is largely the result of flawed design assumptions based on outdated metaphors and paradigms.
My studies seek to explore the ways in which an interface designed to leverage today's increasingly powerful digital devices might provide for more meaningful, personally resonant encounters with the cultural stuff of music. My research is grounded in New Media theory and relies on notions of materiality and embodied interaction to propose an interface aesthetics that encompasses the dialectical relationship between representation and reality, and reintroduces the possibility of an authentic encounter with music's material culture in the digital age.
Study One
The conceit in this study is the identification of user point-of-view with the cinematic presentation of data in the interface. A consistent Albertian perspective and aesthetic realism are set in contrast with behavioral representations that have their own logic, derived from real-world experience, but refashioned into a coherent system. The very act of representing the artifact in a simulated space introduces a material quality to the encounter.
My visual approach in this study references what Apple achieved with Cover Flow; I seek here, however, to accompany the three-dimensional representation with a new set of behaviors that encourage a user to identify with a particular spatiotemporal orientation to the data in the system. The representation is not just a representation — it is integrated with the functionality of the interface. A user is not left to look through the interface at the images inside; the window instead be- comes her perspective as she enters the virtual space of the interface.
Even the preliminary results of this study confirmed my suspicion that a dimensional representation might offer possibilities for cognitive engagement with an interface not afforded by a text-based GUI. But there seemed to be points at which this system limited user agency in such a way as potentially to break the illusion of active engagement.
Study Two
The hindrances imposed on my first study by a limited user agency and inadequate material representation were not immediately apparent to me. My second study instead sets out to address a different set of issues: are there ways in which the cultural stuff of music might be better displayed onscreen?
I opt in this study for an additive method of introducing dimensionality to the album cover. I place a synthetic translucent layer in offset 3D space in front of the album cover, through which a refracted light source slowly paces. This creates a dimensional representation that remains true to the image’s original character, but limits the interface’s ability to create a material impression. As with Cover Flow, a mere representation of space is not sufficient to establish materiality or to support embodied interaction. Nothing about the dimensional strategy of representation in itself transcends the constraints of the screen; without introducing a play that shifts representation into the field of being, an image remains distinct, intangible, immaterial.
Study Three
Sensing, perhaps, that my second study had inadvertently limited the sensations of agency and cognitive engagement I sought to create, I moved in my third study towards evoking space around the album cover rather than within it. I am here attempting to reinforce the idea that the album cover image is a digital “package” containing not only music, but liner notes, photos, and other ephemera.
This third study feels like a less successful retread of some of the ideas I explored earlier. It reveals a couple of interesting new moments, but the interactions and behaviors illustrated do not impart a compelling sensation of embodiment — there is nothing of consequence to learn through acting in this space. It comes across like a three-dimensional version of iTunes LP, and it seems clear its novelty could wear quickly on a user. The study is not very successful at orienting the user’s relationship to the interface or drawing the user into its world.
At this point in my project it became clear that enticing a user into a dimensional simulation would take more than a refashioned image. I would need to push harder on the representational and behavioral strategies I sought to employ to create a sense of agency and embodiment in this simulated world.
Study Four
My fourth study begins by preliminarily sketching an encounter with the larger system — an assortment of images representing perhaps the user’s recent listens or current rotation (the pa- rameters of the entry point of the system I envision will be customizable). I illustrate what it might look like to navigate and explore this subset of items from a collection using multitouch gestures, and I show how the system might be able to configure and represent these artifacts hierarchically according to scale and placement in z-space. Ultimately, these representational and behavioral strategies will be an integrated part of the user’s embodied experience, as they facilitate interpretive and connotative ex- change with the system.
The affordances of the screen-based environment are more successfully allied here with my representational strategies, and a material sensibility does seem to begin to emerge. With each directional gesture, the camera comes to rest at a new location in the composition. The user finds herself in a familiar yet novel space, where the graphic elements from the 2D album cover are activated to reconstruct an experience with a provocatively different resonance. The result, when it works best, is to evoke a simulated space (on its own terms) that refashions prior media into an environment where the groundwork is laid for at least the possibility of new kinds of interaction.
Study Five
The prototype interfaces I modeled in earlier studies overlooked a significant aspect of the material resonance present in encounters with physical artifacts. The images I was using had been scrubbed clean of any visual evidence of their physical presence. The screen’s ability to display detailed information speaks to its materiality through the act of representing intricate levels of physical detail in high-resolution imagery. In the age of the transparent interface, it is common for digital representations of music artifacts to be as interchangeable and nonspecific as the audio files themselves. These representations ideally have no mark of physical presence — they are graphically perfect, pure of taint, abstracted from art that was at one point intended to be printed and live as part of a package.
I believe existing systems’ inability to represent the subtler physical details of remediated artifacts is another factor contributing to the difficulty many have with personally connecting to digital music. One of the things this fifth study seeks to do is reintroduce a detailed level of visual information to digital representations of cultural artifacts, with the intent of evoking a fuller sense of presence in our readings of them. Through experimenting with more thoughtful representational strategies and promoting new interface behaviors in this study, I hope to shed more light on how an interface might impart a fuller sense of agency and consequence within its represented environment.
This study relies on constructing and maintaining an oscillating balance between representation and reality, showing and being (insofar as representation implies distance, a looking through, and reality implies closeness, immediacy, and looking at). This study offers a glimpse of how photographic and filmic realism may play into the aesthetics of an interface that supports a productively ambiguous encounter. But the study plays too fast and loose with its behavioral and representational strategies. The range of possibilities presented to a user needs to be more succinct and evocative.
Final Study
The possibilities and provocations presented by these studies do not imply a one-size-fits-all approach to cultural interface design. Each of the above explorations employs a particular set of representational and behavioral strategies designed to support materiality and embodied interaction. Each holds some promise, but ultimately seems too specific, relying on the particularities of the visual material of the albums in question. A material listening interface needs to accommodate a broad range of artifacts with distinct visual character, while providing an adaptive yet coherent structure within which a user may explore, learn, reflect, associate, read, listen, and play.
These activities suggest an encounter in which representational strategies may shift according to context and user intention. As Suguru Ishizaki describes in Improvisational Design, “A designer’s task...is to anticipate potential changes in the context and specify the communicative forms that design agents should perform according to their immediate situations.” In dynamic information environments, we must look towards developing systems that support forms in flux, content that changes constantly, and experiences that tailor themselves to users as they express their presence over time. An interface that accommodates material listening, then, needs to be both simple enough to invite a user into its logic and robust enough to support an array of unpredictably dynamic artifacts.
My final study is not a revision of an earlier piece, though it builds on some of the most productive aspects of my initial studies. It sketches a system that incorporates representational strategies that could accommodate artifacts with different visual characters — artifacts which, in turn, operate as packages with diverse contents. Interaction with and exploration of these contents supports connotative associations and meaningful encounters for users. The operations and behaviors of the system should allow for a high level of personalization in terms of what is represented (and how).
The culminating study in the series.
All of this is not explicitly enumerated by the animation above, so I will take the opportunity here to say a little more about how I envision this system to work.
As the user encounters the system, she is situated between the artifact last encountered and the rest of her collection. She may continue listening/exploring where she left off, or browse her collection according to other criteria. The study illustrates a user choosing to view “recent” plays, which calls items from a virtual shelf out into an intermediate zone of interaction. Several vinyl 45s and a cassette with a hand-drawn cover are among the artifacts the system presents. The user gestures to see more items, which shuffle themselves to provide new perspectives on relationships and associations that may have gone unnoticed — in this case, she is prompted to think of the late 90s Washington, DC scene when she notices records from her recent Teenbeat kick in the mix with a Slant 6/Make-Up split single.
Not immediately finding what she wants to hear, the user returns all these records, tapes and CDs their place with a swipe. A simulated shelf holds her collection. It rotates into view as each artifact floats to its context-based position. Alphabetical cues indicate approximate positions of selections from the user’s “recent” plays, in place on the shelf, in context with the rest of her collection. If the user chose to view “all” she would see her shelf fill with artifacts. I propose the alphabetical default display is only one among an array of options. For the user truly to feel a sense of agency in the simulated world before her, she should be able to order her collection as she sees fit. (Manual, animated, drag-and-drop arrangement of a collection is impossible in iTunes, and it certainly circumscribes the user’s relationship with her music.)
Next, with a gesture, the user flies towards her shelf as all her “recent” plays emerge in an ordered fashion from their recessed positions. From this orientation, the user can visually explore and easily navigate her collection. Touching and sliding a record or other artifact plays a song or album, and allows the user a better look at the cover if she desires. She can completely remove an artifact from the shelf and rotate it to see the back. Every representation is functional; the interface experience shifts back-and- forth between representing a collection or artifact and being a collection or artifact.
The user can also move into a particular experience with any individual artifact in her collection. At this point, the representation is effectively real — the user is enticed to regard the simulation as a fully present artifact with a productively ambiguous ontological status. Full views of the album’s artwork and sleeve contents are available; so is a potentially infinite array of dynamic associated data. This study illustrates a series of images of The Stooges taken from the publicly-constructed database of Last.fm’s artist page. In theory, the user could access any number of image banks, and even build her own. Any one could be dynamically linked with a particular album or artist.
Information from choice blogs and newspapers, tour schedules, official and unofficial archives, and social networks, for instance, all could be accessed and keyed into a customizable display. The selective, context-based appearance of this data, in time, according to the user’s expressed preference, serves to orient the user in an agency-filled relationship with the interface. The rich associative information helps establish the material dimensions of the listening experience in ways only a dynamically networked experience can. Perhaps in the near future we will begin to grasp an emergent materiality in screen-based experiences with cultural artifacts, based not only on physical cues but also on the particular, customizable experiences users construct and live through embodied interactions with material listening interfaces.
