Integrating Live Support Resources into a Human Factors Validation Test

By Laura Birmingham and Stephanie Larsen 

In a human factors (HF) validation test, it’s important to verify that all available resources intended for real-world use are available to participants to provide a representative user experience and increase the likelihood of a successful test. We consider these types of support to fall into two categories – “passive support,” which includes instructional resources and on-demand videos, and “live support,” which includes interaction with a certain type of person during the test session, be it a customer support helpline representative, a manufacturer representative or on-call support.   

Below, we will first address the “who, what, when, where, why and how” as they relate to incorporating live support into HF validation testing and then how this support might be documented.  

Incorporating live support into HF validation testing 

Why should you implement live support in your HF validation test? 

If live support is something that you will provide as a resource to users in the real world, then you will want your test participants to have a representative experience and benefit from that same support as well.  

What are the different types of live support? 

We have defined three types of live support to focus on in this exploration:  

• Customer support helpline. A phone number that users can call for technical support on the product. 
• Clinical specialist (a.k.a., clinical support, CS, manufacturer representative). A person often employed by the manufacturer who would typically be on-site for initial product use cases, or perhaps all uses of a product. 
• On-call support. A clinician external to the manufacturer who can provide support in person when users reach out as-needed during product use. Unlike a clinical specialist, they would not be present unless called.  

Who should play the live support person during an HF validation test? 

If the helpline already exists (e.g., for a predecessor product) then ideally this helpline could be used in some manner during HF validation testing. Alternatively, someone from the manufacturer, or even the moderator, could serve in this role responding in the same way a helpline employee would. 

The clinical specialist would likely be represented by someone from the manufacturer. This person might also be the same person who provides training to users or could be another individual. If this representative would be available for at least the first several uses of your product – for example, the first few cases with a surgical system -- then it would be realistic to include them in your test.  

On-call support should be a clinician that is hired to serve that role in your test, noting a manufacturer is not explicitly in control of who would provide on-call support in actual use. This type of on-call support would be representative in a case such as a home hemodialysis system, for which users would receive in-depth training perhaps at a clinic, and then would be supported by a nurse for the first several treatments at home, followed by the ability to contact a nurse as-needed. 

Where should the live support person be located during a test session? 

The person serving as the helpline should ideally be remote and not watching the session rather than be in person or observing. That said, this is not always possible, as sometimes it is someone observing or within the session who might need to serve in this role. 

The clinical specialist most likely will be in the room observing, just as they would during actual use.  

For on-call support, as the name suggests, this person should be available on site during the study session but should be outside of the room and can come in to provide support in person as needed, just as they might during actual use. As with the helpline, ideally this person would not be observing the session. 

When to use moderator assistance versus live support? 

A slightly more nuanced question to consider is whether there are times when the moderator should be providing assistance versus the participant receiving live support. Moderator assistance would be needed instead of live support if:  

• The participant does not know about live support. 
• The participant does not want to reach out to live support.
• There is not sufficient time to evaluate all remaining critical tasks and assistance must be provided to move the participant forward. 

How should live support be implemented? 

When considering how to implement live support, keep a few key elements in mind. First, consider how to introduce the availability of live support to test participants. Perhaps this might be done in the training session when discussing what resources are available to support participants once they are using the product independently. Or, if appropriate, the moderator might mention the resource in the test session introduction, such as to inform participants that the clinical specialist in the room is available to support them as would be expected in actual use.  

Next, train and orient live support personnel to usability testing, coaching them to act in a representative fashion and avoid introducing bias. Lastly, consider and plan for the mechanics of having live support interact with test participants, including how participants will signal the need for live support and any boundaries live support personnel should set in their responses, given the goals of HF validation testing.  

Documenting live support in protocols and reports 

Protocols 

Just as you would for any “passive” support or available training, document in protocols that live support is an intended resource for actual use and how it will be implemented. Keep in mind the six questions we covered earlier when considering how to describe this support clearly. 

Also, make sure that you separately document in the protocol how you will implement moderator assistance should it occur to make certain it is defined distinctly from live support. 

Reports 

When it comes time for reporting, we generally take either of two approaches -- documenting the live support among the findings or in a separate, dedicated report section related to instances of live support provided. Most importantly, you want to make certain that live support is documented in some way, particularly if a potential finding led to the live support or if the live support somehow resulted in a finding.  

There is no one-size-fits-all approach to integrating live support into an HF validation test, but if you take into account the considerations in this article, you will be well on your way to doing so effectively.  

Contact our team to learn more about integrating live support into HF validation testing.  

Laura Birmingham is an Associate Research Director and Stephanie Larsen is a Managing Human Factors Specialist at Emergo by UL.