Using Your URRA to Develop Evaluation Activities for Your HF Validation Test Protocol

By Rachel Aronchick and Laura Birmingham 

Human factors (HF) validation testing is a critical step in demonstrating that a medical device can be used safely and effectively by its intended users. An HF validation test’s evaluation activities are based upon the evaluation of critical tasks—those tasks that, if performed incorrectly or not performed at all, could lead to harm (CDER) or serious harm (CDRH) or compromised medical care. Regulatory bodies such as the US Food and Drug Administration (FDA) expect manufacturers to identify these tasks and evaluate them during representative, simulated-use usability testing. But how do you go about developing an HF validation protocol that ensures all critical tasks are accounted for, particularly when it comes to testing complex products or systems with myriad critical tasks?  

The answer lies in your Use-Related Risk Analysis (URRA). Having a comprehensive URRA provides the foundation for developing realistic, naturalistic and comprehensive evaluation activities (i.e., use scenarios and knowledge tasks). In this article, we will explore how to identify critical tasks, leverage your URRA to develop use scenarios for your HF validation test, address gaps through knowledge tasks, and demonstrate traceability for regulators. 

Identifying critical tasks 

As previously described, critical tasks are those that, if not performed or if performed incorrectly, could lead to harm or serious harm. Your URRA should identify all tasks required for product use, as well as the associated use errors / risks, potential harms and severity of those harms. Depending on your definition of potential harm for each severity level in your URRA and whether this product is intended for CDER or CDRH, you will establish a “criticality threshold,” meaning that any risks with a severity of X or higher are considered critical. For products seeking US FDA approval through CDRH, we typically see risks associated with a severity rating of 3 or 4 or higher on a scale from 1 (negligible) – 5 (death) categorized as critical.  

If your product is relatively simple or low-risk, your URRA might include just a handful of critical tasks. However, for more complex systems, such as infusion pumps, dialysis machines or robotic-assisted surgical systems, your URRA could include hundreds of critical tasks.  

While this level of detail is typically a sign of a comprehensive analysis, it can feel overwhelming when translating it into an HF validation protocol. The key is to use the URRA strategically to group tasks into logical, naturalistic scenarios.  

Translating the URRA into use scenarios 

After you’ve identified your critical tasks, you’ll need to translate them into use scenarios. First, start by reviewing your URRA and critical tasks alongside the product’s workflows. Ask yourself: 

• What are the primary workflows? 
• Where are the natural starting and stopping points? 
• How do users typically interact with the system? 

After you’ve answered these questions, document these workflows as initial use scenarios. For example, a dialysis system might include scenarios for priming, delivering treatment, adjusting treatment and completing treatment. These workflows provide the backbone for your protocol. 

Next, map each critical task from the URRA to these scenarios. If a risk occurs naturally and reliably within a scenario (e.g., users must enter a dose as part of treatment delivery preparation), note any additional configuration or supplies needed to evaluate it. However, if a risk does not occur naturally and reliably within a scenario, consider whether you can “make it occur” by adjusting the setup or triggering an alarm at a logical point. If that is not possible, add a new use scenario or plan to evaluate the risk through a knowledge task (discussed below). 

Finally, review your use scenarios holistically and combine or reorder scenarios where appropriate to ensure a natural workflow. This process ensures that all critical tasks are accounted for while maintaining efficiency and realism. 

Addressing gaps with knowledge tasks 

For most products, there will always be some critical tasks that cannot be evaluated during a hands-on scenario naturally, such as understanding contraindications or important storage conditions. In such cases, a knowledge task may be needed to evaluate the risk control measure. During knowledge tasks, you’ll want to pose open-ended questions that assess users’ comprehension or that require users to locate and interpret information independently as they would in their use environment.  

When deciding whether a critical task should be evaluated during a use scenario or knowledge task, consider the nature of the risk. If the task involves physical interaction, such as selecting the correct product from a storage location, a hands-on scenario is likely best. However, if the task is purely informational, a well-structured knowledge task is most appropriate. 

Traceability and documentation best practices 

Developing a robust HF validation protocol that evaluates all critical tasks is only part of the equation. In addition to this identification activity, you must also demonstrate that every critical task has been considered and evaluated; this clear traceability and documentation not only supports regulatory review but also provides transparency into your methodology and shows that you’ve taken a systematic approach to evaluating all critical tasks.  

One effective method for documenting traceability is to create a risk traceability table. This table should link each critical task to its associated risk, harm(s), severity and risk control measure(s), as well as the evaluation method (e.g., use scenario or knowledge task) during the HF validation test. Also, be sure to include success and failure criteria for each task within the protocol to ensure consistent assessment during testing. 

Conclusion 

Although a comprehensive URRA is a key regulatory requirement, it is also the foundation for an effective HF validation test protocol. By using it to identify critical tasks, structure use scenarios, address gaps through knowledge tasks and demonstrate clear traceability, you can design a protocol that meets regulatory expectations and supports the evaluation of a product’s use safety.  

Contact our team to learn more about developing HF validation protocols. Or, sign up for a complimentary account with OPUS, our software platform that provides human factors training, tools and templates. 

Rachel Aronchick is a Managing Human Factors Specialist and Laura Birmingham is an Associate Research Director at Emergo by UL.