We interact with interfaces all the time: our smartphones, the gas pump, online bank accounts. And we all experience times when these daily interactions are confusing, inefficient, or just plain frustrating. But when we’re using medical devices, these interactions can turn into an expense or even a dangerous situation. Consider the following three imagined but realistic scenes.
Scene 1: Three anesthesiologists sit in the lounge talking about the new piece of equipment the hospital recently purchased. “There wasn’t room for it next to the bed, so it had to be out of the sterile field,” Dr. Nash is saying. “We had to bring in another tech to run it and I spent the whole procedure telling him which values to change. I like the technology, but we can’t fit another person in the room.” Dr. Young chimes in, “Remember that emergency case we had in OR 2 last week? Once they moved that new system from OR 6 to OR 2, it took another eight minutes to set it up. Luckily, the patient is okay, but those were eight minutes we didn’t have.” Dr. Stills set down her coffee cup. “Well, there is no way the committee will approve purchasing any more of those.”
Scene 2: Martha, a cardiac device nurse, strides down the hallway with Dr. Marigold. “Last night’s remote monitoring data. You’ll see I’ve written in the blood pressure and weight readings,” she says as she hands him a stack of paper. “Mr. Yang’s ICD delivered two shocks yesterday; I talked to him and he’s coming in for device reprogramming at noon today. Judy Meyer’s spending more time in AFib, as we suspected, so I’d like to turn off those alerts, and Jamal Sani continues to show warning signs of heart failure decompensation, so I think we need to increase his Lasix again or he is going to end up back in the hospital.” Dr. Marigold scans the papers, nodding in agreement. “Perfect. As soon as I finish up with my next patient, I’ll go into the EHR and approve the orders.” Martha nods. She walks back to her workstation, hoping the doctor can approve them quickly so she can get in touch with her patients with the changes before the end of the day.
Scene 3: “Hi, Grandpa! What are you doing here?” Kinley drops her backpack on the kitchen floor and plants a kiss on Jack’s cheek. “Hello, sweetie,” Jack says. “Your mom is helping me with this new app on my phone that controls the insulin pump they gave me last week.” “Cool!” says Kinley as she peers over Jack’s shoulder. “So, Dad,” Kinley’s mother continues her instructions, “Now you press this button that says DELIVER to inject some insulin.” Jack taps the button and stares at the screen. “How do I know it worked?” he asks. “From the beep,” Kinley says. Jack and her mom give Kinley matching puzzled looks.
“What beep?” Kinley’s mom asks. “The beep. Didn’t you hear it? It’s kinda quiet, sounded like a happy little ding,” Kinley explains and then imitates the sound. “I didn’t hear anything,” mumbles Jack while Kinley’s mom furrows her brow and looks down at the paper instructions in her hands. “And, look, the circle changed from red to green,” says Kinley, pointing at the screen. Jack looks closer at the screen. “It did?” he asked. Kinley looks down at her grandpa’s mismatched socks. “Oh, yeah, I forgot, you’re color blind.”
Each of these hypothetical frustrations could have been discovered and remedied long before it reached an actual end user. Companies that are succeeding with users have learned that conducting formative evaluations during product development is an imperative (and increasingly common) best practice to avoid developing medical devices with frustrating—or even dangerous—interactions.
WHAT ARE FORMATIVE EVALUATIONS?
Formative evaluations (or formatives, formative studies, formative research, feedback sessions) include different methods: expert reviews, heuristic evaluations, cognitive walkthroughs, simulated use studies, or user testing. At their core, they are activities that are designed to gather input on a product’s design with the intent to make changes and improvements.
The idea is simple. In simulated use studies or user testing, a person who fits the profile of the intended user (anesthesiologist, device nurse, patient with diabetes) is shown the device, often in the environment in which it will be used, while a moderator steps them through specific scenarios, observing their interactions and asking probing questions about the experience. Then, the product team makes updates to the design based on what is learned.
Conducting formative evaluations during product development is a best practice, but manufacturers often overlook this key opportunity to ensure the safety and usability of their products. Why? Sometimes it’s cost, sometimes it’s time, sometimes it seems difficult to execute. But what will ultimately be more expensive, time consuming, and difficult is responding to a deficiency letter from the FDA or facing lackluster adoption because a product is confusing, frustrating, dangerous, or not a fit for real-life workflows.
There are many excuses to skip formatives. Let’s explore some of the reasons to do them.
WHY CONDUCT FORMATIVE EVALUATIONS?
Understand customers’ use environment and workflows
In scene 1, we hear about workflow issues that affect how anesthesiologists can use the equipment. Dr. Nash gives us some clues about how the form factor limited how the equipment could be used in the room. Dr. Young points out the need for a quick turnaround between cases.
The OR is a crowded place. When introducing a new piece of equipment that needs to take up valuable real estate, the form factor needs to be carefully considered. Does it need to stand on the floor? Will the person operating it be scrubbed in, and have the right level of training to run it? How does it fit in to the workflow of cases? By conducting formative evaluations in a variety of actual OR hospital settings, the manufacturer could have discovered and designed to these requirements.
Understanding the end user
In Scene 2, we meet Martha, a highly trained and experienced nurse. She knows her patients well. She also knows the doctors she supports and what adjustments they make as they monitor their cardiac patients.
Martha’s workflow is interrupted by the need for the cardiologist to take action on clinical decisions she can weigh in on, but must ultimately be ordered by the doctor. She is also printing reports for the doctor to see, and referencing another system to handwrite patients’ vitals. The work that needs to be done outside her applications is just as important to Martha as the data that’s provided within them. By understanding the actual users of this system—the person at the keyboard—and their workflow, the manufacturer can ensure the right data is provided in a useful way and the appropriate tasks are streamlined.
Demonstrating usability
Jack, in scene 3, struggles to use the new app to manage his insulin pump for his diabetes. He’s getting support from his daughter, but the cues built into the design of the interface aren’t noticeable to either of them. “Easy-to-use” graphical user interfaces—particularly for medical devices—can be hard to design. Interactions that a designer believes are clear and straightforward can suddenly be confusing when put in front of a patient.
By observing a variety of patients interacting with the app, the usability issues Jack experienced could have come to light early in the design process. Further, the need for usable interfaces is becoming even more important as patients begin to have more of a say in selecting the devices to help manage their conditions. A deep understanding of the users and the tasks they are trying to complete can go a long way in designing something that’s easy, safe, and even enjoyable to use.
Inform inclusive design
Devices that employ inclusive design are accessible to as many users as possible. Wide curb ramps on sidewalks at intersections may have been originally designed for people in wheelchairs, but they are also useful to parents with strollers or teetering toddlers, people with blindness and a walking cane, travelers pulling wheeled suitcases, and bicyclists at busy intersections. Similarly, medical devices can be designed inclusively when the variety of users and their needs are understood.
Manufacturers may understand the technical role of their users—some may even employ people who used to be in that role—but there’s diversity within every user population. An anesthesiologist in a small community hospital has different needs from her peer in a large teaching institution; the nurse who spends all day reviewing device data has different needs from her peer who also sees patients in a clinic; a person who uses an app to manage a condition has different needs from one who is also experiencing comorbidities, or cognitive or physical impairments.
Workflows, standards of care, medical judgements, understanding of terms, eyesight, dexterity—the list of ways a user population can vary goes on and on. While a manufacturer can’t possibly uncover all this variation, conducting research with a mix of users will reveal needs across a more diverse group of users.
The FDA expects it
Medical device governing bodies have recognized the importance of demonstrable usability and safety. The FDA has issued guidance and IEC has issued standards encouraging manufacturers to implement usability engineering practices. While these guidelines primarily focus on safe use, they also encourage manufacturers to uncover and resolve usability issues.
The FDA will expect to see a human factors report with a submission. Among other things, this report must outline the formative testing that was conducted, what was learned, and what changes were made from the findings.
This report also includes the results of summative (or validation) testing. While this is a different type of evaluation from a formative, it is the manufacturer’s opportunity to demonstrate safe use of the final device. In addition to uncovering issues, formative evaluations can serve as a dry run for the more rigorous summative testing, reducing the chances that it will need to be repeated.
Summary
The worst time to find out a product has usability and safety issues is when it’s already on the market. The best time is before the product design is complete, when there is still an opportunity to fix them.
There is no substitute for observing the intended users of a medical device interacting with your product. Taking the time to do formative testing early in, and throughout, product development goes a long way toward ensuring safe, usable devices that are cleared or approved, and adopted.
About Fathom Consulting
Fathom Consulting is a business and design consulting firm that takes a human-centric approach to the design of products, services, and operations. We have a 24-year legacy of bringing the key external and internal voices to the table to create alignment, build momentum, and collaboratively and confidently meet customer needs.
With deep expertise in human-centered design and a rich understanding of human factors engineering, we ensure products are designed with users’ needs at the forefront. Our areas of practice support clients as they examine concepts, implement device design, test and train users, and prepare for FDA submission.