Evaluation

During the product development process, effective design evaluation is central to user interface design and the drive to ensure a great user experience.

There are many ways to evaluate a product and the best way can depend on the stage of product development as well as the available resources (e.g., time, funding, prototype or finished products). Our evaluation methods include:

Early Stage Evaluation

We employ many techniques to collect feedback on early design concepts. One technique is called a cognitive walkthrough. The approach may call upon respondents to study product sketches, simple physical models, or a series of static computer screens and think through and explain how they might perform a task.

Examples:

  • Evaluate the workflow associated with operating an insulin infusion pump by viewing multiple sample screens and handling a “block model.”
  • Obtain feedback on multiple concepts for the soft goods (e.g., backpack, fanny pack, messenger bad, satchel) associated with a wearable heart pump (left ventricular assist device).
  • Assess user preferences among multiple patient monitor design concepts using 3D rendering software plus accurately weighted, 3D-printed models.

Middle Stage Evaluation

Frequently, we are called upon to critique a product, also referred to as an expert review or heuristic analysis. In these cases, we apply established design principles and our judgment to identify user interface design strengths and opportunities for improvement. Such evaluations can usually be done quickly and at low cost yet produce valuable findings.

We are also asked to conduct formative usability tests of products. Such tests can be quick to deliver findings for a design team that might be in the middle of development sprint. Alternatively, a formative test might be a more formal activity involving detailed test planning and reporting.

Middle Stage Evaluation

Examples:

  • Perform an expert review of the screens associated with a fitness app to be used on a smart phone.
  • Conduct an 8-participant formative usability test of an exoskeleton used to help people suffering from stroke regain their stride.
  • Conduct a 15-participant “pre-summative” usability test of a dialysis machine, following a test plan (i.e., protocol) intended for use in the subsequent summative (i.e., validation) usability test.

Product evaluations expose design strengths and opportunities, such as those shown in the following table for a hypothetical home dialysis machine.

Sample Findings

Positive

Negative

Visual alarms are displayed in a conspicuous (i.e., attention-getting) manner.

[Source: Critique]

Smaller and weaker users are likely to have difficulty lifting the 40 lb. machine.

[Source: Critique]

Users considered the set-up workflow to be intuitive.

[Source: Cognitive walkthrough]

Several users were confused about the meaning of the symbol on the backlight control.

[Source: Formative usability test]

Test participants with small hands and those with large hands felt that the handle provide a secure, comfortable grip.

[Source: Formative usability test]

Four out of 12 test participants were unable to install the tubing set correctly.

[Source: Formative usability test]

14 out of 15 users practiced aseptic technique when connecting the patient line to the machine.

[Source: Summative usability test]

Two participants kinked the drain tube when placing it in its holder.

[Source: Summative usability test]

Late Stage Evaluation

When a product is nearly final, we can conduct a summative (i.e., validation) usability test. For medical products, this is a pivotal activity that can produce proof that a product can be used safely and effectively by the intended users. Such testing is also a regulatory requirement for products seeking FDA clearance. Such testing might take place in a conventional usability test laboratory or a more advanced setting, such as a high-end medical simulator.  

We have conducted such tests with small (n=15) and very large (n > 200) population samples. Our test plans are routinely reviewed and tacitly accepted by the FDA. We conduct tests in our own laboratories, in facilities worldwide, or in rented focus group facilities. As required, our test reports thoroughly describe the usability testing approach, test findings, and root causes of any persisting interaction problems. We base our testing and reporting approaches on available guidance from multiple sources (e.g., FDA, AAMI, IEC), as well as best practices

Examples:

  • Conduct a 75-participant, summative usability test of an insulin pen injector to generate data proving that the product can be used safely and effectively by adolescents, adults, seniors, caregivers, and healthcare providers. (Note: Sample size is dictated by FDA requirements pertaining to validation studies. See: Applying Human Factors and Usability Engineering to Medical Devices, issued by FDA on February 3, 2016)
  • Conduct a 15-participant, summative usability test of a heart-lung machine used by perfusionists to sustain patients undergoing open-heart surgery.
  • Conduct a 12-participant, summative usability test of a tablet application used by automobile mechanics to diagnose and, where possible, fix electrical and mechanical problems.

Post-market stage evaluation

We frequently conduct product assessments, including design reviews and usability tests, of marketed products. Such reviews may serve the following purposes:

  • Provide benchmark performance data that forms the foundation for developing a next-generation product or competing product.
  • Provide user interaction-oriented performance data upon which to base a marketing claim.
  • Gain insight into reported interaction problems that might have led to product complaints, returns, and other types of failures.

Examples:

  • Conduct a usability test comparing the perceived ease of setting up a dialysis machine to deliver a treatment.
  • Conduct a study with people who have diabetes to determine which lancing device is easiest to use and causes the least pain.
  • Perform an expert review of the user interface of a recalled surgical device to determine the root cause of use errors that led to harm, and suggest design changes as part of a CAPA effort.