User Evaluation in the Absence of User Data with Platform Devices

Human factors studies are essential to identify and eliminate any risks posed by drug delivery devices to the intended user. This, in turn, helps create user-centered designs that encourage patient adherence. Human factors guidance and international best practices, therefore, recommend that testing is performed on intended users, highlighting the importance of having test participants that reflect real end-users of the device and the therapy it contains. Targeted testing creates confidence in the results and their applicability to the wider user population.

With platform devices that may be deployed across a range of therapeutic formulations, however, the intended therapy area and intended patient profile is not yet known. Accurate identification of the intended user population is not an option here. The device could be used by patients with different conditions and people who display different characteristics and levels of physical and cognitive abilities. Furthermore, manufacturers will need to consider that either a healthcare professional, caregiver, or a patient may use the device.

Rather than simply testing the device on a specific target patient group, an inclusive strategy will be needed, wherein device safety and effectiveness are tested across a broad base of potential end-users. In order to meet the needs of all potential user groups, study samples will need to be carefully selected to ensure they are sufficiently representative of the potential patients. This will enable product designers to make informed decisions about the device, while also assuring business partners that any usability-related risk-factors have been uncovered and dealt with at the early stages of development.

A good starting point for manufacturers will be to decide on an appropriate sample size to carry out the tests and demonstrate a thorough examination of the device and its use environment. Ensuring wide representation enables timely identification of use issues and relevant mitigations before development has gone too far. At the early stages of development, it is considered good practice to include five to eight participants per distinct user group in usability tests. The number of distinct user groups, however, may vary and is likely to be higher for platform devices given the unspecified intended user.

Once five subjects have been tested, any major usability issues should have been detected, and the amount of additional information to be gained from adding more subjects starts to decrease. To demonstrate, one study showed that doubling the sample size from five to 10 subjects only increased the mean percentage of usability problems identified from 85.6 percent to 94.7 percent (Table 1). In the later stages of development, prospective customers will seek confirmation that their intended user has been thoroughly examined throughout the design process and any iterative user testing. U.S. and U.K. regulators recommend that 15 to 20 participants per user group are used for validation testing. This figure is deemed large enough to account for differences between users within the user population.

If each user group is composed of five to eight people in the early stages, the next step is to determine how many user groups are needed to cover the wide range of user characteristics and needs for platform devices. At Owen Mumford, we’ve developed a sound framework, which suggests dividing subjects into seven user groups (Table 2) to cover the range of characteristics that could influence the device’s usability for end-users.


The sample plan includes four categories of people who may use the device or aid the patient in using the device, namely children and adults, caregivers, and healthcare professionals. The remaining three groups cover users’ capabilities in using and interacting with the device, known as perception, cognition, and action (PCA). For optimal results, it helps to make each use-impairment group mutually exclusive to demonstrate the impact of different aspects of the device interface on each user group. For prospective customers, this may be extremely helpful and a decisive factor in choosing a device.

In order to make the sample group even more representative, the user groups above can be further broken down to include secondary characteristics such as gender, ethnicity, and hand dominance. One example of this could be to include people with biomechanical as well as neurological impairments within the seventh “Action” group (Table 2). Wherever errors or difficulties are encountered, this may help to identify the root cause.

The challenge that comes with user evaluation plans is determining the right number of studies, the most appropriate stage of development to carry out studies, and the appropriate level of prototype fidelity. This is made more complicated in the case of platform devices given the lack of data on intended users. An effective sampling strategy is therefore critical to cover all bases and assess risk across the range of possible users. While a user evaluation sampling plan can be preceded by a less formal test with easy-to-access participants—such as members of the company—using a wide group of representative participants is recommended as early as possible.

Using the human factors sampling strategy outlined in this article can serve as a valuable framework to anticipate the needs of future customers and meet best practices in the most cost-effective fashion.

This sampling strategy may also be adjusted to the manufacturer’s specific needs. For instance, user groups may be larger to allow for greater diversity. Likewise, group sizes can be adjusted for commercial reasons, in order to align with the projected needs of the business or demonstrate a minimum representation of a specific comorbidity. 

---

A highly experienced Human Factors Engineering Manager, Finola Austin boasts 15 years’ experience in mentorship and management of human factors services in safety-critical industries. Her career began in Occupational Therapy within acute, long term, and community settings, and her training in accessibility has given her special insight into the needs of impaired users. Since then, Finola has successfully planned and delivered Human Factors activities for hundreds of handheld medical devices, including auto injectors, emergency use devices, inhalers, injection pens, and lancets, and is proficient in the generation and review of documentation. She has executed numerous user evaluation studies in the United Kingdom and United States—including studies on safety engineered devices, injection pens, and color differentiation.