I handled this project from Human Factors International, and we worked together with Seagate one of the world’s largest hardware manufacturers to help them strengthen their online presence in four different countries.
Test Objectives
:: Observe and record participants’ reaction to the usability of the site designs
:: Identify visual, navigational and architectural issues that need to be addressed
:: Understand how users in localized regions think about and interact with the site
:: Analyze findings, prioritize issues in terms of impact, and provide concrete recommendations
:: Observe and record participants’ reaction to the usability of the site designs
:: Identify visual, navigational and architectural issues that need to be addressed
:: Understand how users in localized regions think about and interact with the site
:: Analyze findings, prioritize issues in terms of impact, and provide concrete recommendations
Why did we choose Remote Usability Testing?
We went ahead and did RUT against the traditional lab testing because of the following reasons.
:: The test required participants from widely dispersed locations
:: The test required a large number of participants and
:: There were costs and/or time restraints that limited face-to-face testing.
:: The test required a large number of participants and
:: There were costs and/or time restraints that limited face-to-face testing.
However there are also limitations of RUT i.e., We lose an opportunity to observe the non-verbal behaviors.
Challenges
The most critical things were to streamline the project activities, manage a large team in various locations, and ensure efficient logistics. The process which evolved as a result was helpful to execute similar projects effectively and efficiently across different time zones, different languages involving a large team. It was imperative to have a process in place that can be replicated across countries, and allows for successful and timely completion of the project.
Remote Usability Testing setup
As you can see in the above image, other than the actual field experts (UT experts) it involved many other resources.
Test Protocol - in three languages
We initially crafted the protocol in English, and translated it to other languages based on the countries tested. Wanted to make sure nothing is lost in translation. So, we had one translator to do the translation from English to regional language, and a different translator to translate it back to English. This technique worked beautifully well even if it means some extra money spent on translation.
We initially crafted the protocol in English, and translated it to other languages based on the countries tested. Wanted to make sure nothing is lost in translation. So, we had one translator to do the translation from English to regional language, and a different translator to translate it back to English. This technique worked beautifully well even if it means some extra money spent on translation.
Test sessions
We gave each session the same effort and importance in order to gain more qualitative insights. We also had a log that was thoughtfully designed to log the observation so it also was easy to make a quick findings of the study. Having the log sheet well organized made the analysis easier.
A deeper analysis of the findings lead to interesting cultural insights where users behaved differently for the same given tasks. This offered scope for understanding user behavior and mental models in different cultures.
What did the RUT report cover?
:: What users liked globally.
:: What users did not like globally.
:: Problems users faced globally (along with severity for each problem), while performing the tasks.
:: Problems users faced only on the regional sites, while performing the tasks.
The report also covered design solutions for important and frequent tasks, that dramatically improved the performance of the site.
Key results
:: Drop-off reduced by 50%
:: Satisfaction increased by
