Test Management

Reducing Medical Device Risk with Usability Testing: The Why, the How, and the Who

By |

This is a guest post from Christopher Kim, M.D., Manager of Human Factors & Usability at the product development firm, Bresslergroup. It originally appeared on their blog. 
    Industry research shows that each year, approximately 400,000 hospitalized patients experience some kind of preventable harm.

    Medical device and equipment flaws caused by poorly researched design, mishandling, user error, and malfunction are common causes of medical errors. Many of these errors can be attributed to lack of standardization, poor design, poor maintenance, differences between devices from different manufacturers, and more.

    Devices recalled nationwide are typically taken to task for usability issues that could have been prevented with proper usability testing throughout a product’s development lifecycle. This means there’s a significant opportunity to reduce the potential for harm by conducting quality usability testing early and often.

    The Need for Usability Testing

    Digging further into causes of those medical device recalls, the top four, according to industry statistics, are:

    • Software issues
    • Mislabeling issues
    • Quality issues
    • Missing the mark with specifications

    By testing early enough in the product design lifecycle, life-threatening medical device errors can be prevented by the appropriate design changes. Unfortunately, testing is often done too late in the process when the design has already been “frozen.” Once you get to this point, use errors are often forced into mitigation by changing the label or altering the instructions-for-use (IFU). As the statistics show, these measures are frequently insufficient.

    So how can you prevent this by proactively starting a user-centered design process?

    1. Begin: Create a Human Factors Engineering and Usability Engineering Plan

    First, draft a strategic Human-Factors Engineering (HFE) and Usability Engineering (UE) Plan. Putting your users at the center of your product design process may seem intuitive, but once you get into the weeds of development, these user needs can get lost.

    It’s helpful to keep three questions in mind: Who are the users? What are the product’s intended uses? In what use environment is the product being deployed?

    From there, dive deeper:

    Who are the users?

    • Who are all the intended users?
    • Do any of these users interact only with specific aspects of the user interface? Could there be a subset of users who interact with only one aspect of the device? Could there be another clump of users who interact with another portion entirely, with very little interaction in between? For example, take software that’s been designed to help patients manage their sleep apnea and CPAP (continuous positive airway pressure) treatment while also allowing physicians and nurses to review their settings and compliance. A nurse, physician, and patient might all interact with the software in very different ways. Although the nurse and physician might have similar interactions, it’s important to determine whether or not certain actions are appropriate for the nurse or the physician, or both.

    What are the product’s intended uses?

    • What’s the intended use? How and in what circumstance should the product be used? What is the intended medical indication the product is meant to treat?
    • Are patients using the product to self-treat? Could there be any physical limitations that prevent a certain patient from interacting with your product as intended?
    • Is training required? What level of training is required, if necessary? (We highly recommend training be kept front of mind throughout the development process.)
    • What is the user interface of the product? This includes all points of interaction — physical and digital — between the user and the device.

    In what use environment is the product being deployed?

    • Is the product intended to be used exclusively in the home by a patient? In an emergency room or ambulance?
    • Does the product use differ, depending on the environment of use? Whether the user is in an adult ICU, a NICU, or in an ER trauma bay — each of these contributes to significant differences in the way a product is used.

    From a regulatory standpoint, the FDA will be looking to see that all elements of the users, uses, and use environments have been researched and incorporated into the plan for conducting proper usability testing in the relevant simulated-use environment with the intended users and uses.

    The ideal process might include multiple rounds of formative usability testing, which can be used to inform a Use Failure Mode and Effects Analysis (uFMEA). As you go through the cyclical design process — adding design inputs, testing them, refining, then adding more design inputs — you’ll eventually reach a point where you’re comfortable with the product you’ve created.

    A final production-equivalent medical device should be put through validation, or summative, testing, to assess whether or not the product can be deemed safe and effective to use.

    2. Assess: Create a Use Failure Mode and Effects Analysis (uFMEA)

    A uFMEA assessment is meant to identify components of the user interface and the impact of task failures. What are the possible failures that could occur? Why would these failures occur? What are the consequences of the failure and what’s the associated harm?

    Start by creating a step-by-step list of all tasks, required to accomplish the device’s end goals. (Tasks are defined as the action or set of actions performed by a user to achieve a specific goal.)

    The uFMEA also notes what could go wrong if any of these tasks are not completed correctly. To help determine this, go through a PCA (perception, cognition, action) analysis process. As your team goes about defining tasks, determining failure modes, and assigning levels of severity of harm, you may next want begin to categorize your tasks as critical versus non-critical. Critical tasks are those which, if performed incorrectly or not performed at all, could cause serious harm to the patient or user where harm is defined to include compromised medical care. User tasks are often tied to product requirements, and usability testing (formative and validation) will be needed to prove these user needs have been met.

    Finally, when developing your product’s uFMEA, make sure your team has thought through the level of mitigation that will, ultimately, be required by the IFU and labeling within the context of intended use. During this process, you may also come to a point where you recognize whether or not your product will require some level of training.

    Be diligent about identifying all potential use errors and outcomes before getting into validation or even pre-validation testing. Be as clear as possible about what could go wrong and what needs to go right to merit a task has been completed successfully. These success criteria will feed into a task analysis table that will define what needs to be tested from a usability standpoint.

    And finally, keep in mind that a uFMEA should be treated as a living, breathing document that evolves as the product progresses through the design process. Expect it to change as the device goes through rounds of formative usability testing, is put in the hands of users, and as the design team gains a better understanding of how the product can and should be used.

    3. Research: Use These Three Approaches To Usability Testing

    There are three usability testing approaches to consider:

    Rapid Insight Testing may be appropriate if all that’s needed is a quick touch-base to keep the design grounded in user needs. Roughly five to six participants are recommended, but it’s possible to test fewer in certain situations. This type of testing typically occurs prior to entering design control.

    We recommend conducting Rapid Insight Testing on all elements of the product, including human factors. Sample questions are: How did it feel to use or handle the product? Is the grip intuitive or did someone need to show me how to grip it? Are we testing all of the possible users of the product or are there others that could be seen as a potential user? This is your opportunity to evaluate all tasks, not just the critical ones. Additionally, this is your chance to get as much subjective feedback on the product as possible! What are their preferences? What are they used to seeing and what would they like to see changed?

    Formative Testing begins to inform the design process. Anywhere from five to ten representative users per user group are recommended and aim to simulate the use of the product in the simulated use environment with a high-fidelity prototype. We suggest conducting a minimum of two to three rounds of formative testing during a product development cycle. This is what’s going to lead to product design change mitigations, which is ideal — better now than later! Design changes are hugely preferable to changing labeling or instructions-for-use (IFU) late in the game.

    Validation Testing is required by regulatory bodies in the U.S. and Europe, and elsewhere around the world. At this stage, testing with a production-equivalent product is expected, and when dealing with FDA, 15 representative users per user group is required, as is representative simulated use in the simulated environment.

    It’s important to note that once you’ve reached validation testing, only critical user tasks should be evaluated, so it is important to evaluate all non-essential user tasks earlier in the process via rapid insight and formative testing.

    4. Comply: Resources and Tips for Regulatory Consistency

    A long discussion could be had when it comes to regulatory compliance, but here are a few resources, tips, and misconceptions to keep in mind.

    Resources for Regulatory Compliance

    The Association for the Advancement of Medical Instrumentation’s (AAMI) standard HE75:2009 provides general considerations and principles to help manage the overall risk of use error with best practice design elements and integrated solutions. This document acts as a comprehensive reference for how to incorporate human factors engineering into the medical device design process.

    As far as IEC 62366 goes, it guides the entire usability engineering process, including the elements of accompanying documentation and training. The main purpose of this document is to help define the human factors engineering and usability process as it pertains to medical device design, including consideration of risk management.

    The FDA guidance — 2016’s “Applying Human Factors and Usability Engineering to Medical Devices” — lays out the needs and expectations for your Human Factors/Usability Engineering Report.

    The FDA developed this guidance to “assist industry in following appropriate human factors and engineering processes to maximize the likelihood that new medical devices will be safe and effective for the intended users, uses, and use environments.” The Center for Devices and Radiological Health (CDRH) considers human factors testing a valuable component of product development for medical devices, so it is very important to be mindful of usability throughout the design process.

    You Say FDA, I Say IEC

    It’s good to know some of the key differences in expectations between the FDA (Food and Drug Administration — U.S.) and the IEC (International Electrotechnical Commission):

    • The FDA expects a single report — the HFE/UE Report — to hold all relevant information while the IEC 62366 requests similar information and data but does not require a single report.
    • The FDA expects to see conclusions at the beginning of the document. The IEC does not have these same expectations.
    • IEC 62366 uses the term “summative study,” but the FDA uses the term “validation”; the terms are interchangeable, which confuses those who are not familiar with the industry. With that in mind, note that “usability validation” is a very specific element you need to test for that is within an overall umbrella of design validation.
    • The FDA only accepts validation usability studies conducted in the United States; IEC does not state such requirements for location. Note, however, that any differences in the intended uses, users, and use environments between the U.S. and Europe and any other international body needs to be appropriately tested.

    Don’t Go It Alone!

    We know this is a lot to take in. If you need a development partner, Bresslergroup is always up for a good challenge. With our design strategists, mechanical and electrical engineers, and industrial and interaction designers, we’re well-equipped to assist in making a better product. (Read about our medical product design experience and expertise.)

    In addition to providing design and process recommendations and collaborating on system enhancements, we can plan, conduct, and report on all elements of your usability testing needs — from rapid insight testing all the way through validation.

    Hear experts from Bresslergroup and Jama Software discuss strategies for mitigating design-related problems with medical device development by watching our joint webinar, “Accelerate Medical Device Development While Reducing Risk.”

    Latest posts by Christopher Kim, M.D., Bresslergroup (see all)