Case study: Usability testing of a health worker tool developed to aid in the identification of people at risk of experiencing negative health outcomes
Context
Medic partnered with a community health organization to study the feasibility and
efficacy of implementing predictive algorithms on a health worker digital tool to deploy resources towards people at risk of adverse health events. On conducting program monitoring evaluation, the team observed that the task completion rates were consistently low compared to the projected events. The team decided to conduct a study to investigate why this was happening as well as gather insights into the health workers and clients experiences with the intervention.
efficacy of implementing predictive algorithms on a health worker digital tool to deploy resources towards people at risk of adverse health events. On conducting program monitoring evaluation, the team observed that the task completion rates were consistently low compared to the projected events. The team decided to conduct a study to investigate why this was happening as well as gather insights into the health workers and clients experiences with the intervention.
Shield icon used visually depict individuals at risk of negative health outcomes in the app
My role
As the lead user researcher and experience designer, I worked closely with 1 data scientist, 3 project managers (external and internal), 1 UX designer, 3 researchers (external and internal), 1 data analyst and 2 software engineers.
Timeline
The user research activities took 4 weeks; 1 week dedicated to quantitative analysis of the events generated and completed by health workers, 1 week to plan the research with the team including developing participants selection criteria, interview guides and data analysis templates, 2 days to recruit participants, 1 week to conduct research and 1.5 weeks to analyze data and compile findings.
Research Goals & Objectives
Goal: To find out if predictive models deployed to health workers tools were working as intended and establish health workers and community members perceptions of the intervention.
Research objectives:
1. To investigate how health workers were identifying high risk individuals in the community and in the digital health tool.
2. To establish whether health workers were visiting at risk individuals flagged by the digital tool and how often that happened.
3. To investigate whether health workers were using visual cues in the digital tool to quickly identify profiles of high risk individuals.
4. To establish the perceptions and attitudes of household members flagged by the digital tool as high risk.
5. To assess the knowledge, perceptions and attitudes of health workers towards the predictive models and high risk individuals identified.
Research methodology
The study employed a mixed method research design (used both quantitative and qualitative methods). Quantitative analysis, led by the data scientist and analyst, was conducted to determine the expected health worker intervention tasks, analysis of the telemetry data, number of tasks generated on health workers devices, number of tasks completed and an analysis of the socio-demographic variables for individuals flagged as high risk. Three (3) interview guides and 1 checklist were used during the information gathering sessions which included observations, in-depth interviews (4) and focus group discussions (3).
Recruitment criteria and process
The participants selection process was informed by the data analysis outputs. A purposive sampling approach was employed to select clients who had been flagged as high risk individuals and the health care workers assigned to their households, all stratified across gender, wealth quintiles and education levels. The same sampling technique was used to recruit individuals who the health workers believed had been missed by the predictive models for a validation exercise. Participants recruitment was conducted by the field teams ahead of the visit.
Fieldwork planning
The team held sessions to align on how to conduct the activities and roles assigned to the different groups. During interviews, two note takers were assigned for each session and other team mates encouraged to also take notes and complete the observation checklists. At the end of the day, the team convened for debrief sessions to collate and process findings from the sessions.
Analysis and synthesis process
At the end of each day, the team convened for a debrief session to collate and process findings from the sessions. Each team shared their findings and highlights from the sessions were documented on sticky notes and worksheets. In addition, the team transferred the field notes to an excel data tracking template.
After fieldwork, the teams reconvened to analyze the findings. Post it notes were put up on the wall, grouped and emerging themes identified. From the themes, the team created actionable insights which reflected the user needs and desires of how predictive models and associated service workflows could aid in care delivery.
Outputs/Deliverables
The findings revealed that there were software bugs in the code deployed to health workers devices. The team established that health workers were experiencing challenges using the application and that they were not using the developed cues and features designed to aid in the identification of high risk individuals. These findings guided data driven iterations of the system design, service workflows and informed retraining needs. Improvements were prioritized (framework employed: impact, device performance and level of effort) and deployed incrementally and frequent feedback sessions conducted to gather feedback on the improvements. Overall, task completion rates improved and health workers felt confident executing their duties and communicating information to high risk individuals. Reports and power point presentations were produced for different stakeholders.
Reflections
1. This exercise demonstrated the importance of utilizing both quantitative and qualitative data to solve problems.
2. Resulted to improved working relationships with the data team.
3. Frequent feedback sessions ensure that bugs are caught early.
4. Retrospective sessions held served to identify ways of mitigating the issues/ code bugs.
