When and Why to Conduct User Research Studies, Formative Usability Testing and Human Factors Validation Testing

By Ashley Zabroski and Katelynn Larson

At first glance, user research, formative usability testing and human factors (HF) validation appear to be variations of the same activity. But treating them as the same thing can be a costly mistake in product development. In this article, we discuss the when and why of conducting different types of formative evaluations and usability studies for medical devices and the regulatory imperative for these key activities. 

User research studies: A strong foundation for product development

User research is the earliest and often most influential HF activity because it reveals the real-world needs, motivations and constraints that should shape every subsequent design decision. 

While some insights can be gleaned from desktop research, uncovering the day-to-day workflows, experiences and niche needs directly from intended users can highlight key opportunities for product development that may be obvious to the intended users but completely outside the perspective of product developers. Alternatively, an idea that a product developer thinks will be revolutionary might complicate existing workflows for intended users, making the product less relevant and hindering adoption.

Some common forms of user research include:

• Surveys – In the modern era, surveys are often administered online and allow researchers to collect quantitative data from a large sample to identify trends and preferences. However, they lend themselves to multiple-choice or short-answer questions, limiting the depth of their insights.
• Interviews and focus groups – A small-group or individual interview session can enable researchers to investigate key points of interest further and allow interviewees to elaborate on their perspectives. These methods typically involve a smaller sample size.
• Contextual inquiries or observations – These research methods involve firsthand observations of intended users interacting with a similar product in the actual use environment. These observations enable researchers to better understand workflows, team dynamics (if applicable) and the impact of the use environment on product use.
• Diary studies – In a diary study, a user is asked to record data over several weeks or months. Temporal trends can emerge from the collected data, providing additional opportunities for product improvements.

Typically performed during the early stages of product development before a user interface has been finalized, well-executed user research can:

• Clarify findings from initial desktop research
• Identify gaps and limitations in existing solutions
• Help prioritize potential design features based on user needs and preferences
• Provide context on use environments and workflows
• Serve as the foundation for user needs and design requirements
• Provide data to guide stakeholders and key decision-makers in selecting design concepts

Once user needs and the context of use are well understood, product developers can begin evaluating how early design concepts perform in the hands of intended users. Developers can then turn to formative usability testing for further insight into how users interact with products.

Formative usability tests: An opportunity to refine a product’s design

Formative usability testing is essential for iteratively refining a device and identifying usability issues before they become costly to fix, potentially requiring additional testing, stakeholder alignment on design changes and delays in bringing the device to market. It puts product prototypes in front of users to see how they interact with the product. Although early formative tests might look similar to user research, their goals differ: user research identifies needs, and formative testing evaluates how well early designs meet those needs.

The focus of formative usability testing can vary significantly, from the product in its entirety to a single component. This type of testing can be performed at nearly any stage of product development before the design is finalized for validation.

• Early-stage formative usability tests can take several forms and can overlap with user research activities. For example, a developer might ask a participant to provide feedback on several device prototypes or concepts to determine which options users prefer and what features promote safe and effective use. Because the device design is fully in flux, the expectation is that data from the formative test will inform the overall device design. Early-stage evaluations help the product developer choose the design direction before investing in detailed development. 
• Mid-stage formative usability tests might be completed with a prototype that developers are confident in, but the labeling, instructions for use (IFU), training plan or other elements of the user interface might still be in development. The objective of mid-stage formative usability tests is typically to refine the product to enable safe and effective use. For example, a drug-device combination product developer might conduct a differentiation study with several dose strengths of the product to establish whether labeling is clear and users can accurately select the correct dose strength. 
• Late-stage formative usability testing can take the form of an HF prevalidation test, where the test format and setup follow the same general method as the HF validation test. Such studies enable researchers or HF specialists to identify potential red flags that could hinder a successful HF validation test, giving the developer an opportunity to make any final changes ahead of the full HF validation test. Late-stage formative testing can also be used to determine whether a product is suitable for a different market. For example, if a developer were bringing a product from the United States to the European Union, data from a formative test can inform what changes (if any) would be required for the additional market. 

Formative usability tests are largely at the discretion of the developer. As such, the format, scope, evaluation activities and even test conduct can vary widely depending on the desired goals of the usability test. 

The iterative refinements informed by formative evaluations help developers more confidently enter an HF validation test with a final design that has already been optimized for safety, usability and clarity. 

HF validation test: Evaluating a product’s safety and efficacy

The U.S. Food and Drug Administration (FDA) and other regulators expect developers to provide data supporting the safety and efficacy of medical devices and combination products for use by intended users in the intended use environment. HF specialists collect the data required by regulators during the HF validation test (i.e., summative usability test). 

Unlike formative usability tests and user research studies, for which regulatory guidance is less prescriptive, regulatory bodies, including the FDA, have more specific expectations for the HF validation test, ranging from the evaluation activities to the user-group selection. According to the FDA, an HF validation test must:

• Represent actual use scenarios – Participants are asked to work through simulated use scenarios they would perform during actual use, such as conducting a procedure on an anatomical model or completing an injection into an injection cushion. Use scenarios encompass the entire range of product interactions, such as preparation, administration and disposal of a product, as well as realistic training sessions if a user would be trained on the product prior to use.
• Test a production-equivalent device – Although the expectation of formative tests is that the device is still in development and subject to change, HF validation tests must include the final device design. Any changes made to the user interface after the HF validation test must be justified or retested to demonstrate that no additional use errors are introduced. 
• Account for critical tasks – Critical tasks are those that, if performed incorrectly or skipped during product use, could cause harm (including compromised medical care) to the patient or user. Critical tasks are defined by evaluating the product’s use-related risk documentation (e.g., use-related risk analysis/use failure modes and effects analysis), and all identified critical tasks should be included in the HF validation test’s evaluation activities.
• Include representative users – Participants with the same indication, job responsibilities or characteristics as the intended users should be included in the sample. When it is not feasible to include actual users, such as for a drug product for an extremely rare condition, proxy participants with the same performance-shaping characteristics (attributes that influence the way an individual interacts with a device) as the intended user group might be acceptable. For example, the FDA specifies that each distinct user group must include at least 15 participants.
• Represent actual use environment – The impact of the environment on device use can be significant. As such, the testing environment should reflect relevant characteristics of the intended use environment, generally by using an appropriate simulated test environment. For example, an environment testing a device used during surgical procedures might have bright lighting, props indicative of an operating room and a soundtrack simulating appropriate background noises.

After testing, HF professionals conduct a thorough root cause analysis of all findings to determine their causes. Data collected during an HF validation test, combined with a residual risk analysis that determines whether the remaining risk inherent in the product is acceptable, supports the claim of safe use.

Regulatory imperative for usability testing

Regulators have fewer specific requirements for formative evaluations and more specific expectations for HF validation testing. However, a strong user-centered design typically performs best during HF validation testing, and robust formative evaluations provide developers with key insights that inform a user-centered design. As a result, performing both formative evaluations and HF validation tests during the product development process can strengthen a regulatory submission, potentially reducing regulators’ feedback on submission documentation and the need for remediation activities to backfill missing data.

Reach out to Emergo by UL’s HF specialists to learn more about different types of formative evaluations and usability testing, the benefits and drawbacks of different approaches and how to integrate them into your product development process. Or sign up for a complimentary account with OPUS, our team’s software platform that provides human factors engineering (HFE) training, tools and templates to complete these and other HFE activities.

Ashley Zabroski is a human factors specialist and Katelynn Larson is a technical writer at Emergo by UL.