A Practical Guide to Formative Product Evaluations: Why and When to Use

By Emilee Stanczyk and Johanna Thommai 

When developing a new product, an important step for medical device and pharmaceutical manufacturers is to seek a thorough understanding of the product’s intended users and their needs when considering where and how they will use the product. Conducting formative research in the early stages of product development is a powerful tool that helps gather insights into your target users and their needs and preferences, supports early-stage design concepts and prototypes, and uncovers potential usability issues early on. This insight enables designers and engineers to make informed decisions that may save time and reduce costs. Let’s take a look at specific types of formative research and how and when to apply them. 

User research  

In the early product development phase, the most important questions are: Who are the product's users, and in what context will they use it? During this exploratory phase, conducting user research can provide insight into user needs, beliefs and challenges they encounter that might influence the way they interact with a product or their expectations for how it will function. For example, how do users currently accomplish similar tasks, and what challenges or frustrations do they face when doing so? What needs must your product meet to appeal to your users? What constraints or conditions affect how users might interact with your product? User interviews enable teams to answer questions on a wide range of user-related topics and empathize with the user. With this understanding, you can tailor your product’s design to best help the intended users.   

Let’s imagine you are developing an injection device intended for patients with Parkinson’s disease to self-inject or receive an injection from their caregiver at home. During the exploratory research phase, you might learn that current injectable treatments are difficult for Parkinson’s patients to administer due to common disease symptoms, such as tremors or slowed movement, impacting device handling. With this knowledge, you can make informed decisions regarding the device’s shape, size and suggested grip that might be better suited for patients with impaired dexterity.   

Cognitive walkthroughs  

What if your product is still in the design phase and not yet ready for use? A cognitive walkthrough is the perfect approach if you’re seeking quick input on a low-fidelity prototype, specific elements of the user interface or early concept design sketches. The benefit of using lower-fidelity stimuli typically keeps participants more focused on the core workflow for completing tasks instead of focusing on the aesthetic details. These types of evaluations typically consist of researchers asking participants to describe how they would complete various tasks using the product and share real-time feedback on how the product facilitates desired workflows so you can better understand participants' perspectives as they encounter the design for the first time.  

Cognitive walkthroughs enable you to discuss many topics in a short amount of time, making them particularly helpful when seeking input on multiple design elements. These evaluations can also be conducted with product designers on standby to create sketches or modify prototype concepts behind the scenes, such as from a private observation room, in response to participant feedback. The updated concepts can then be shown to participants in real time or used in subsequent participant sessions.

Heuristic evaluations   

Another goal in the early stages of product development is to uncover key usability issues before full-scale product manufacturing begins, when design changes can be time-intensive and costly. One effective method is to invite a team of experts — individuals with human factors engineering (HFE) experience, user experience (UX) training or external consultants — to review your product against relevant design guidelines and requirements (also called “heuristics”) to identify user interface issues.  

During a heuristic evaluation, evaluators conduct a walkthrough of the interface guided by the heuristics, highlighting any deviations or pain points and enabling designers to address immediate concerns. The Nielsen-Molich usability heuristics are among the most widely used theoretical frameworks for evaluating usability and include 10 core principles, such as Visibility of System Status and Error Prevention.  

Guided by expert judgment and relevant established design principles, a heuristic evaluation is an effective method for identifying usability issues early in the design process.

Surveys and focus groups    

Surveys and focus groups both allow you to get feedback from multiple participants at once, but each has its own benefits and drawbacks. 

Surveys and questionnaires are effective tools for gathering opinions from large numbers of participants or obtaining input from geographically dispersed audiences with minimal oversight and moderation effort. Online questionnaires can be a convenient way to validate exploratory research hypotheses or crowd-source quantitative data to identify trends and preferences. However, they may be less effective when deeper qualitative feedback or detailed insights are needed. For example, if you’re wondering what type of delivery system users would prefer or which of two concepts they find most visually appealing, a survey is a good option. However, a focus group may be more appropriate when you’re seeking deeper, more nuanced feedback that explores factors such as emotion, motivation and context. This type of research can help reveal which features users want in a delivery system or which design elements they find visually appealing and why. 

Like surveys, focus groups enable researchers to collect many opinions relatively quickly, but they also give participants an opportunity to provide detailed feedback, ask and answer clarifying questions, and share insights that may not emerge through an online questionnaire. In addition, focus groups allow researchers to ask impromptu questions as new topics arise and adapt the discussion based on participant responses. 

Focus groups can also provide valuable insight into group dynamics, such as how health care professionals (HCPs) in different roles interact with or defer to one another. Another benefit is their ability to spark new ideas, as participants often build on one another's comments and perspectives, leading to observations that might not emerge in individual responses.

Formative usability testing    

Formative usability testing incorporates elements of the other evaluation methods together for a more thorough, standardized evaluation method. The primary goal of a formative usability test is to uncover specific pain points, confusion and inefficiencies in the current interface, ideally weeding out the usability issues through iterative rounds of testing. This can be done by having participants interact with the product to complete realistic use scenarios and answer questions to understand the root causes of any usability issues. 

Formative usability testing can — and should — be conducted at multiple product development stages, from early-stage evaluations with low-fidelity (but functional) prototypes to later assessments focused on fine-tuning a narrower set of features and addressing any remaining usability issues before heading into HF validation testing. When conducted in the later stages of development, these usability tests can also help inform the validation test methodology, including task prompts and interview questions. 

Of course, no product will ever be perfect (nor will its users), but formative evaluations conducted early and often throughout the development process can significantly improve the user experience. Understanding your users and how their prior experience and performance-shaping factors may influence product use is a key input to designing a safe and effective medical device that meets users where they are — aligning with their needs, expectations and real-world environments. Investing in formative evaluations early can help identify and address usability challenges before they become more difficult and costly to resolve. 

Contact our team to learn more about how formative product evaluations can support your product development efforts. You can also sign up for a complimentary OPUS account to access HFE training, tools and templates.

Emilee Stanczyk is a managing human factors specialist, and Johanna Thommai is a human factors specialist at Emergo by UL.