USABILITY TESTING
Test the mobile app of AsthmaMD through Heuristic Evaluation, Cognitive Walkthrough, Usability Testing and Comparison Study.
TEAM
Nick Alonso-Emanuel
Sonali Bhurke
Yangye Cao
Dean Christopher
Yvonne Nillissen
CHALLENGE
AsthmaMD is a free mobile application for users to quickly and easily log their asthma activity, medications, and causes of their asthma in the form of a diary. AsthmaMD logs all of this activity and converts it into reminders, action plans and charts that can be shared with their physicians and included in medical records.
STRATEGY
AsthmaMD differentiates itself from the competition in its gathering of anonymous asthma data. It allows users to opt-in/out for anonymous tracking of key information such as severity of asthma attacks, triggers, time, date and geolocation data. Aggregating this information provides researchers with unprecedented data and insight into the causes and external correlations of asthma. AsthmaMD is the only mobile application awarded the prestigious UCSF Collaborative Research Network (CRN) grant, and is available on both iOS and Android.
HEURISTIC EVALUATION
The AsthmaMD mobile application was evaluated with the Heuristic Evaluation method developed by Jakob Nielsen. Nielsen’s 10 general principles for interaction design are used as broad rules of thumb to approach usability problems within an application. Findings are rated on a severity scale of 0 to 4, where 0 is a problem not recognized by all evaluators, and 4 is a problem that is imperative to fix.
METHOD:
Discovery was conducted by a team of 5 evaluators who were experienced in user testing, but who were not product experts in asthma care. They spent between 2 to 4 hours each, working with all aspects of the application. Following is one sample of the Heuristic Evaluation Summary with descriptions of findings, levels of severity, heuristic principles effected, and suggestions for improvement. Overall, critical usability problems were found in the areas of new user data entry, content display, design consistency, general information availability, and Action-Plan functionality.
Example from AsthmaMD mobile app Heuristic Evaluation Summary with descriptions of findings, levels of severity, heuristic principles effected, and suggestions for improvement.
COGNATIVE WALKTHROUGH
The purpose of the cognitive walkthrough is to examine the application’s usability for target users. Five major tasks were evaluated using this method. The process flow and actions were evaluated to determine if they answered the usability questions and meet usability requirements.
METHOD:
Each evaluator compiled their findings and highlighted design strengths and issues encountered during the walkthrough. As ease of learning is a major goal of a cognitive walkthrough, the task of creating and starting an Action Plan was identified as a key area of focus for this report.
Example of Cognitive Walkthrough task and actions evaluation.
COGNITIVE WALKTHROUGH RECOMMENDATIONS
ACTION PLAN
ACTION PLAN
AsthmaMD is not easy to use for creating a new plan, editing created plans or starting a plan. The user should not be required to enter the same data for all zones without being aware of the reason and significance in tracking their medication usage. The user would be wasting time and prone to make errors because of repeatedly adding same information for same medications. AsthmaMD should allow for easy and standard methods to edit a plan so that the user can quickly change medications according to their prescriptions and conditions. The user should be allowed to change the medications added, delete and make changes to the related dosage information after creating the action plan. Currently,
AsthmaMD provides the Edit plan button is on an unconventional path, which can be frustrating for a user.
RECOMMENDATION EXAMPLE: Provide Edit Plan and Delete Plan features on the page which allows user to create a plan for ease of access. Edit functionality should use methods to allow addition of new medication and deleting them from the plan. The app should provide information on the reasons for showing a Start button only on the Yellow Zone. If starting on Yellow Zone is suggested in all cases, then the user should be brought on this page after finishing all steps for creating a plan. The user should not have to search how to start his/her plan by clicking on the Green/Yellow/Red Zone tabs.
USABILITY PLAN
EXECUTIVE SUMMARY
EXECUTIVE SUMMARY
In this study we observed, interviewed and surveyed participants to evaluate the usability of the AsthmaMD app, to understand the difficulties users encounter when executing upon core tasks. Our goal is to evaluate the app in its current state and provide recommendations to the developers for making the app easier and more pleasant to use.
We conducted five usability tests to better understand the usage, navigation and flow of AsthmaMD. Our usability test focused on five tasks that cover the core functionality required by users of the AsthamMD app. These tasks include: 1) Registering for AsthmaMD: 2) Setting up an Action Plan; 3) Setting Medication Reminders; 4) Editing an Action Plan; and 5) Finding Video Tutorials and FAQs.
METHODS
Purpose
The AsthmaMD application has been launched by concerned physicians and is being used by many people who suffer from asthma, mild to severe. The product team is concerned that while adoption of the app was swift when released through a network of physicians, usage has been declining and developers suspect the interface may have problems. Based on an initial heuristic evaluation and cognitive walkthrough, they would like to gather objective data and get recommendations for improvement.
Test Objectives
1. Can users efficiently register their personal data, including simple prescription medications?
2. How quickly and easily can user set up their individual Action Plan?
3. How successful are users in setting reminders?
4. How successful are users in editing an Action Plan after it is created?
5. How successful are users in finding video tutorials about using the AsthmaMD app?
6. Do users agree that the AsthmaMD icons and symbols represent recognizable actions?
7. How satisfied were users with the ease-of-use of AsthmaMD?
Tasks
The participant will be asked to perform 5 tasks on a mobile phone with AsthmaMD pre-loaded. The tasks for the participants to complete are as follows:
● Add their personal data into the app, including their simple-prescriptions medications.
● Create their individual Action Plan (as given)
● Set a reminder to take a medication at a certain time of the day (within 10 minutes of current time? To see if it comes through?)
● Return to Action Plan and make an edit to medication.
● Find a video tutorial about how to use AsthmaMD
Example of task instruction:
Imagine that you have been recently diagnosed with asthma. You have been given an Action Plan by your physician and rescue and control medications. Your physician has recommended that you track your symptoms, and follow your action plan, and suggests using the AsthmaMD app. You download the app and now you need to enter your personal data.
Post-Task Satisfaction Survey
The participant will be asked to complete a post-task survey. The moderator will present the participant a printed survey with the following questions:
● On a (Likert) scale between “Very dissatisfied” and “Very satisfied”: How satisfied were you with the ease-of-use of AsthmaMD overall?
● On a (Likert) scale between “Strongly Disagree” and “Strongly Agree”: Do you agree that the AsthmaMD icons and symbols represent recognizable actions?
Evaluation Measures
We will be collecting both qualitative and quantitative data to measure the user’s performance based on the test objectives. We will be looking the following basic measures:
● Success rate of a user completing a task
● Time to complete a task
● Errors committed (navigation and selections)
● User's’ subjective satisfaction
Here are the questions to collect the metrics:
1. Was the user successful in registering for AsthmaMD app?
2. What questions did the users ask for entering the initial medical data? Were they able to easily enter their personal data and medications?
3. What questions did the users ask as they work through creating an action plan?
4. How do the users feel about how long it took them to create an action plan?
5. What did the users feel about editing the action plan? Were they successful in making changes to their action plan?
6. What were the difficulties that the users faced about setting up reminders?
7. What paths did the users take when they wanted to know more about any feature on the app?
8. How well does the site support the tracking and monitoring functionality for a patient? Does it assist the user in reminding about medications and following their action plan?
RESULTS (example used Task 1)
Task 1: Can users efficiently register their personal data, including simple prescription
medications
Our first task involved the users going through the on-boarding process. They were tasked with entering demographic information and information relevant to the app. This included entering name, email, and gender as well as asthma specific information such as PSP code of their peak flow meter, their medications, and flow data.
FINDINGS AND RECOMMENDATIONS (example used Task 1)
Task 1: Users are often confused by the Registration interface, adding unnecessary time
to task.
Participants liked the overall look of the registration interface, finding the simplicity of the design appealing. They generally understood the contents being requested in the input fields and forward arrows directing them to input additional information.
However, quantitative results show some dissatisfaction, and qualitative results show that
difficulty with the registration interface occurred in several micro-interactions.
● First we learned that users were looking for a way to “save” their entry and finding none.
The “Back Arrow” is not a standardized way to save an input. Said a participant, “Yes. I was not sure if the DOB and Height was saved or not. And I didn’t know if it was supposed to or not.“
● Participants also were confused by the request for a PSP code, losing time and patience
while trying to resolve the request in the input field. Said a participant, “What is PSP code?
Can I go on? Let’s look on Google!”
● And when participants attempted to enter data, such as simple prescription medications,
the registration interface obscured the listing of controller medications. Said a participant,
“It was confusing!! why is rescue med first? should come after controller. Scroll down not
a good way to find.” And another participant noted, “in medications section, cannot easily
get back to edit/revise the input, feel lost in the middle of action.”
Finding and Recommendation graph for Task 1 of usability test. See illustration of these findings below:
The usability data for Task 1 as shown in report appendix. Qualitative data was also gathered. See below.
APPENDIX
Includes recording consent form, session script and forms, pre-test questionnaire, and final questionnaire.
COMPARISON STUDY ASTHMA MD APP
EXECUTIVE SUMMARY
Compare which of two mobile interface styles, a Back-Arrow or Save Button, is more efficient for users in registering their Peak Flow Meter data into the AsthmaMD application. A key issue that emerged from original testing was the lack of sufficient user feedback in the application, specifically during the registration process.
In this study we conducted a comparison study to determine which feedback mechanism is better suited for affording users the ability to save data that they input, and then to confirm their changes have been saved. The study was presented using two hi-fidelity prototypes, once using a Back-Arrow for saving information, the other using a Save Button. The current AsthmaMD app utilizes the Back-Arrow functionality. Our goal is to evaluate both interface styles, to determine which is more efficient, and which better affords the user the understanding that their information is being saved.
Four separate comparison tests were proctored by each of the five (5) team members. Each comparison test lasted approximately 5 minutes, and included the following: Time for educating participants about the test; reviewing and signing a consent form, facilitating the comparison test of entering peak flow data, and answering some short questions thereafter.
Based on our comparison test, and after conducting statistical analysis, we’ve determined that an interface change is warranted. By utilizing Condition B, where a Save button replaces the back arrow, users were much quicker to save their information, and did so knowingly without confusion or uncertainty due to a lacking feedback mechanism.
PURPOSE
We will conduct a comparison test between two data-input interactions:
● Gather quantitative data on both interactions.
● Gain insight into specific interaction efficiency
● Gain insight into efficiency of similar interactions throughout the app
● Create a usability study protocol
TEST OBJECTIVE
Compare which of two mobile interface styles, a Back-Arrow or Save Button, is more efficient for users in registering their Peak Flow Meter data into the AsthmaMD application.
METHODS
To understand user's response on the two conditions of registering the asthma peak flow data and to analyze the test results, we will conduct a between-groups comparison test to collect the time on task for each condition and develop a T-test to determine the test significance. Five experimenters will conduct the comparison tests individually by testing two participants per condition and gathering a total of 20 data points. The study will test on one dependent variable: Time on Task with two conditions as follows:
● Condition A- A 'back arrow' saves the user's newly entered Peak Flow number in the Android version of an AsthmaMD mobile app.
● Condition B- A 'save button' saves the user's newly entered Peak Flow number in the Android version of an AsthmaMD mobile app.
Each participant will be asked to sign on the Informed consent form before the test, then
presented with a hi-fi prototype for Condition A or Condition B and given the task to input and save their peak flow data. The experimenter will monitor the whole test process and use the stopwatch to record the completion time. The time on task will start once the participant clicks the numbered keypad and stop once the participant selects the arrow or button.
MATERIALS
The prototypes are created with Axure and designed with a small subset of screens with limited functionality. The link and screens are as follows ( Figure 1 ).
Each experiment will record the data of time on task and input all the data points into a googlesheet for the further statistical analysis. T-test will be used to compare and analyze
the test results to determine if a significant difference in time-on-task between completing
Condition A or Condition B
● The Null Hypothesis: There is no significant difference between time on task for Condition A and Condition B.
● The Alternative Hypothesis: There is a significant difference between time-on-task for
Condition A and Condition B.
STATISTICAL ANALYSIS
Our participants were split into a between groups study with half our participants (n=10) using the back button to save data, and the other half (n=10) using a save button to complete the task. We measured time to complete the task as our dependant variable.
We conducted a t-test to compare two groups of participants (those saving data by clicking the back arrow and those saving by clicking the save button). We found a significant difference (t (18) = 2.10, p = .008), such that participants who used the save button were able to complete the task faster (M = 4.06, SD = 8.64, 95% CI [0, 11.32]) than participants who used the back arrow to save (M = 18.97, SD = 23.98, 95% CI [0, 39.13]). Based on this analysis we are able to reject the null hypothesis that there will be no
difference in time on task between the back arrow and the save button as conditions to save data.
FINDINGS AND RECOMMENDATIONS OF ASTHMA MD COMPARISON STUDY
The AsthmaMD mobile application has a back-arrow button which saves any entered data, but our previous analysis had found this to be a key design issue. Users’ had no feedback that their medical data was indeed saved. As a group we redesigned the page used to enter a Peak Flow Meter data by adding a clear “call-to-action” button – SAVE. We also positioned it on the right hand top corner for instant visibility and mapping to the mental model of any user. Based on the results of the t-test conducted we found that the participants who used the Save button (Condition B) could complete the task faster than
participants who used the back-arrow (Condition A) to save the data. We observed significant difference in time-on-task between completing Condition A versus Condition B.
Using the interface style of Condition B, we could see that all the participants’ were able to quickly save the entered medical data. Please refer to the Methods section for the pictures showing the Save button on the prototype. They could locate the Save button and save their data efficiently.
Similar usage of back-arrow button for saving has been observed in executing several important tasks throughout the app. The problem pertains to navigation, lack of a call-to-action button, improper labelling of the buttons (e.g. iPhone versions use ‘Enter Data’ near the back-arrow). All of these problems can be resolved by placing a Save button on the page where medical data is entered by the user and using back-arrow to work as a Cancel button or for taking the user back to the previous page.
Participant data from comparison usability test and T-test output. This chart data was contained in the appendix.
APPENDIX OF ASTHMA MD COMPARISON STUDY
Appendix to the Asthma MD included data shown above, recording consent form and script.
