Framing/Reframing: Adobe AR Residency
a write-up of my final experiment for the Adobe Aero residency, Session 008.
My final experiment for the Adobe Aero AR residency is called Framing/Reframing. The experiment is a culmination of the residency in that it builds on a lot of different forms I worked with during the 12 weeks of the residency, and captures some themes I am interested in generally. Specifically, in this experiment I hope to explore modes of offering new ways of thinking about your physical surroundings and the social arrangements that exist within them. Whether or not it succeeds at that, I think I found some ways to think about AR that I'm excited to share.
First, I'll talk about the formal experimentation. In my work I try to connect with the specific affordances of a new technology. Affordance can mean many things, but I tend to think of it as what specifically a new technology asks and permits a user to do. What are some things that Augmented Reality, and in particular mobile AR with Aero, affords? Some answers that emerged through my experimentation were:
creating a unique kind of video by using AR as an on-the-fly compositing tool
experiencing presence and agency through interactivity and encouraging physical movement
having the same virtual experience exist in multiple physical environments through sharing
I'm going to present how these come into play in Framing/Reframing as if they came to me in a linear way, but for the record my *actual* process was more a series of loopy epiphanies.
AR as an on-the-fly compositing tool
Throughout the first part of the residency, I looked at mobile AR as a unique type of video making tool that makes compositing the CG imagery into the real world easy, producing a new type of video. This made sense to me because, in my opinion, mobile AR is not actually super immersive. For one, it's awkward to hold a phone up, especially in public, where you appear at best to be taking a video of nothing, and worse to be taking a video of someone you don't mean to. Beyond that, it's on a seriously distracting device (ungh, phones), and there are plenty of opportunities for something to screw up the viewing angles. But! In the hands of a clever cinematographer the *video* of a mobile AR experience can look magic. And, especially during my time as a resident (quarantined due to the pandemic) we experience each other's lives through videos we share. So, making videos using Aero felt right to me.
AR, interactivity, agency, instructions, and movement
Interactivity is one way to allow a user to have agency, and ultimately agency makes an artwork more clearly a direct *experience*, which in my opinion is impactful. Agency exists in a few forms in this Framing/Reframing - the user chooses where they want to see this piece (their physical environment), what they want to frame, which of the vignettes they want to experience in that environment, when to trigger each new piece of audio, and how to move through the experience physically. While I set up certain parameters - like the sounds, environments, and code - the user makes choices on how the work will play out.
One thing that is still unclear to me - and maybe not that important - is whether the "artwork" is for the user of the .real file, or that the .real file is really more of a tool, for the user to make an experience for someone else (in the form of a video). Maybe it's something in between, a collaboration between the user and me.
I've used instructions as "part of the work" in the past, as a way to create an interactive experience that avoids unintended confusion in the user, and have found that instructions can actually intensify the feeling of presence in an experience, in the same way that a script allows an actor to access a particular emotion within themselves. Instructions as art has a long history, and I think instructions can play an interesting role in AR art experiences in particular. I want to cite a project by Adi Dahiya that did this brilliantly, in which he developed a set of "interactive performance pieces" in the form of instructions for how to use Zach Lieberman's AR app Weird Type. In that way, Adi was a choreographer of an experience. In this experiment, I included a set of poetic instructions, to balance that it is both an experience for the user, and a tool for making an experience (in the form of the video output) for someone else.
AR and "the real world"
AR, as opposed to other interactive media (like websites or video games) or other digital spatial media (like VR) is situated in the surrounding physical space. So, throughout the residency, it was important for me to connect the virtual scene with the surrounding environment in direct and intentional ways. For most of the residency, this meant making videos that took place in my apartment (since it was during stay at home measures in New York). But, Aero in particular seems to be best suited to making experiences that can be shared. Sharing these means that ultimately, as an artist, I have no actual control over where it is experienced. How could I resolve these two things? By making something which frames the environment - any environment - and enfolding that variability into the work. I hope that the work takes on a different meaning based on where the user places it: what does this frame here? what do these words mean here?
So, that's the long, non-linear story of how I landed on this form: a set of instructions + a .real file that you can use to meditate on a physical space, and then perhaps also use to create a video to share with others to contemplate on their own. In a way, the format isn't so different from a face filter, but instead of your face, it's the world.
It's as much about your perspective as it is about mine.
Why?
There were many influences that led to this exploration, but more than anything it was the feeling brought on by the pandemic and simultaneous political uprising, which has opened up for many (including myself) new ways of seeing how the interconnected systems in this world function: how they perpetuate themselves, who they serve or harm, how they can change.
I was hugely influenced by a book my friend and collaborator Marina Zurkow had shared with me, Ideas-Arrangements-Effects, which I am still processing continue to be inspired by. The crux of this framework is that ideas are embedded within arrangements, which in turn produce effects. These elements function in various loops, and can provide a way of looking at the world that allows you to identify ways that it can change, or that you can change it.
Ultimately, whether or not this experiment successfully encourages deep contemplation and change, I hope that you (whoever is reading this) recognizes that you have agency. You are an agent of change in this world!
Instructions for Framing/Reframing
(kind of a poem):
Find a place with an arrangement
Frame the arrangement in an empty shape
If you'd like, record
Take several deep breaths
Touch the empty frame
If you'd like, read out loud
If you'd like, answer the question
What does it frame?
Move through it
Think about the question
When you pass through it, touch it
If you'd like, read out loud
Are these questions statements?
Pass through the last frame
When you do, touch it
Listen for loops
Take several breaths
If you'd like, turn around
See where you were
Touch the question so it follows you
Do what else you feel is right
