Hands-Free Control in VR
Testing a prototype
Testing a prototype
Note: The full text of the published study report is available online here.
Background
Traditional controllers in virtual reality (VR) can be cumbersome and distract the user from immersion into the VR experience. Hand-tracking technology, which enables the user to interact with objects in the VR world simply using their bare hands, has the potential to revolutionize the way we interact with virtual environments. It was has medical potential, in VR-based training, assessment, and rehabilitation: The VReha project, for example, has found success using hand-tracking with VR to assess spatial-cognitive functioning.
Oculus Touch controller, the traditional VR interaction solution
The Problem Space
Most people have little prior experience with hand-tracking technology, and may not be used to interacting with objects without the sensation of touch. The sensors used in this system are less precise than those used in handheld controllers. This presents a user experience challenge: How can we develop a hand-tracking controller so that they are intuitive to use, and less prone to the types of errors that arise from the technical limitations of the system?
Hand-tracking prototype hardware: Leap Motion sensor affixed to the Oculus head-mounted display
The Project
Our study aimed to assess the usability of a prototype hand-tracking VR controller, based in Leap Motion (now UltraLeap) technology, developed by the Fraunhofer Institute for Telecommunications. To determine the effectiveness of the Institute's modifications, we compared the prototype to its more simple, “out-of-the-box” hand-tracking cousin. Additionally, to get a sense of how the prototype stood up to existing standards for hand interaction with VR, we included a handheld Oculus Touch controller.
Hand representation in virtual world. Top row: unmodified, out-of-the-box system, (A) outside grabbing radius, (B) within grabbing radius, and (C) grabbing the object.
Bottom row: modified prototype system, (D) outside grabbing radius, (E) within grabbing radius, and (F) grabbing the object.
Modifications
In development of the prototype hand-tracking system, the following modifications were made to the basic, out-of-the-box version:
- extended grab radius to turn "near misses" into hits
- semi-transparent hand representation to increase visibility of the target object
- a small spotlight emitted from the hand when an object was within grabbing range
- a required open hand gesture before grabbing
Top row: study conditions, showing the view of the virtual environment. (A) unmodified, out-of-the-box hand-tracking system (B_Leap), (B) hand-tracking prototype (HHI_Leap), (C) Oculus controller.
Bottom row: study procedure. Each participant completed a training round, followed by a grab-and-place task, then finished with a questionnaire, for each device (within-subjects design).
Research Questions
1) How does the modified Leap system compare in performance and subjective experience to the unmodified version?
2) How does it compare to the traditional controller?
Predictions
Prototype vs. Unmodified
1) The modified hand-tracking prototype would have better performance and more favorable subjective ratings across the board than the basic hand-tracking system. Why? The modifications were aimed at reducing the number of accidental pickups and drops of objects in the virtual environment.
Prototype vs. Traditional Controller
1) The prototype hand-tracking controller would provide a more intuitive and pleasant user experience than the traditional controller. Why? We could not find many prior studies from which to make this prediction, but we made this prediction based on our own experiences with the controllers, and under the assumption that the free-hand interactive qualities of hand-tracking would make it more intuitive to use.
2) The traditional controller would have a better performance than the prototype. Why? Because its design would be familiar to most people (a video-game controller with buttons and a joystick); the hardware limitations of the hand-tracking system would create performance issues (i.e., obstructions of the single front-facing camera); and the because the user experience of the traditional controller, after many rounds of product testing by Oculus, would likely be more polished than that of our prototype.
A view of the experimenter screen and a participant during the study (shown with permission).
Metrics
Performance: accuracy, time on task, # of accidental drops
Subjective experience: questionnaire rating items
Tools
Leap Motion, Oculus Rift HMD, Oculus Touch controller, Unity, Excel, R
Task
32 participants completed a task in a virtual reality environment: a cube would appear on a table, the participant would pick the cube and try to place it onto a target area as quickly as they could, then repeat the process with a new cube. The task was completed 10 times, for three different sized cubes, for a total of 30 times. Participants completed this task with each of the three controllers (within-subjects design) in a randomized order.
I used the R statistical software to conduct the analysis so that it would be reproducible and transparent. You can download it here if you want it.
Results
Prototype vs. unmodified hand-tracking
The modified hand-tracking prototype showed a clear improvement in the rate of accidental drops over the unmodified, basic hand-tracking solution. We also set benchmarks and established a paradigm for comparing a prototype hand-tracking controller to a typical handheld controller. In other words, other hand-tracking prototypes can be compared to this one if they undergo the same paradigm.
Accidental drops. For each device, individual participant scores at left, overall device means with 95% confidence intervals at center, and smoothed distributions at right. B_Leap = unmodified hand-tracking; HHI_Leap = modified hand-tracking prototype; Oculus = traditional controller; *** = p < .001 (paired-samples t-tests)
As expected, the traditional, handheld controller provided the best performance (speed, accuracy, and error rate).
An unexpected result, however, was that participants rated the Oculus as equal or better along all measures of subjective experience. Particularly surprising was that they rated the Oculus as feeling equally natural and intuitive as the hand-tracking controllers.
Subjective Usability Scale (SUS), a much-used questionnaire designed to test any sort of interface. Both hand-sensing prototypes received nearly identical overall usability ratings. Each device shows individual participant means at left, overall device means with 95% CI at center, and smoothed distribution at right. B_Leap = unmodified hand-tracking; HHI_Leap = modified hand-tracking prototype; Oculus = traditional controller; * = p < .05
SUS: Brooke, J. SUS-A quick and dirty usability scale. (1996) In Jordan, P.W., Thomas, B., McLelland, I. and Weerdmeester, B.A. (eds) Usability Evaluation in Industry. Taylor and Francis, London: 189-194
Scoring margins: Bangor, A.; Kortum, P.; Miller, J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Stud. 2009, 4, 114–123.
We believe this is because people were more familiar with traditional video game controllers, and because the lack of haptic feedback, while otherwise using the hand as it normally would be used, might have created a creepy, "uncanny valley" feeling in participants. However, on every dimension, participants rated the hand-tracking prototype as acceptable: within 1.1 standard deviations of the center rating. This is a pretty good showing for an interface that many people were using for the first time.
Ratings of subjective experience (means and 95% confidence intervals of 5-point Likert scale). B_Leap = unmodified hand-tracking; HHI_Leap = modified hand-tracking prototype; Oculus = traditional controller.
Take Aways
- Our study indicates promise for hand-tracking in VR-based tasks such as moving virtual objects, the primary gesture required of VReha's spatial memory assessment task
- We furthermore showed support for our modifications (see above) as a way to improve / reduce the number of accidental drops in a pick-and-place task
- The dimensions of familiarity and haptic feedback should not be overlooked when considering the usability of a hand-tracking system (or any novel interaction system) compared to the traditional solution
Credits
I took a lead role in the design, data collection, analysis and writing of the manuscript for this user study. Mustafa Lafci handled the programming of the paradigm in Unity (the VR software engine). Michael Gaebler, Paul Chojecki, Detlef Runde and David Przewozny supervised the project and offered invaluable feedback and support.
For a more detailed description of the project, read the published study or check out the repository on GitHub.
