Human Factors Activities at a Safe Distance

In the medical device and pharmaceutical industries, human factors engineers rely on engaging prospective users to learn about how products are — and will be — used. Medical devices and combination products typically require validation testing to provide evidence that a product is safe and effective for use. Participants are often recruited to complete realistic training (sometimes in groups) and perform simulated use scenarios with the product being evaluated. These studies are often conducted in market research facilities, usability labs, or medical simulation centers. Representative users, such as patients with chronic illnesses or healthcare professionals, must be represented in studies because their unique characteristics mean that they may interact differently with products than individuals without the same experiences, knowledge, or biophysiological state.
 
Due to the current pandemic, it may not be possible, let alone advisable, to engage in group activities or even to travel to public spaces, such as market research facilities. It goes without saying that simulated usability testing cannot — and should not — be performed as usual. Because of this, pushing out human factors engineering activities has begun to delay medical device development, and many program timelines are being derailed. But it is certainly possible to adapt human factors activities to keep medical device development on track.
 
We can do this by taking novel but scientifically viable approaches to adapt our activities and focus on those that are typically conducted remotely. Simulated use testing with a moderator and participant may be considered the ‘industry standard’ approach to collecting data from end users (and historically, this is required for validation testing), but other effective methods of research, risk management, and development activities do not necessitate face to face interaction. These are the activities that Farm, a Flex company, is working with our customers to execute, in an effort to make safe and efficient progress, all from our newly inhabited home offices. Curious to know what can be done while keeping a safe distance? We discuss some of the highlights below. 
 
Competitive Landscape Research
For a product to truly succeed, medical device developers need to understand products that are already on the market. It’s important to develop robust user profiles to determine how existing devices on the market are being used, the effectiveness of those devices, and where the competition falls short. If a key function is absent or difficult to use, developers of the new device have a clear path to product differentiation and gaining competitive advantages. To identify the most compelling opportunities, user feedback — not just desk research — must be collected through thoughtful study design. Although it may seem easy to identify features that are absent from competitive devices using simple reviews of instructional material, the impact of those features is largely unknown. Without proper research, you could invest time and money into developing a function, only to discover during formative studies that most users don’t value the update, because it doesn’t substantially benefit the patient, nor the users’ experience. Strategic data, on the other hand, can pinpoint design changes that yield tangible improvements, increased user adoption and market success.
 
Generative Research 
Generative research shapes the concept of a product by understanding market landscapes, user needs, “white space” in the industry, predicate products, potential use cases, environmental features and limitations, and any currently standard processes or workflows. Generative research is absolutely crucial for developing a new or next-gen medical device. Luckily, there are plenty of ways to conduct generative research remotely. Desk research, online surveys, phone interviews, video tours, and diary studies each provide us with answers to the ultimate generative question: What should our product be? 
 
Semi-structured phone interviews are a great place to start gaining information and determining the right questions to ask. These interviews include strategically designed questions, but the natural flow of the conversations may uncover additional information that would not otherwise be collected. It is helpful for semi-structured interviewers to have a deep expertise in moderating and probing on medical topics, so that when a conversation naturally swings to a relevant and novel topic that should be further explored, the moderator can understand and facilitate the conversation. The findings from phone interviews, as well as the generative efforts listed above provide valuable insight to drive design decisions and give your new product a competitive advantage in the market.
 
Formative Research 
Some combinations of product and formative research objectives allow researchers to conduct usability testing remotely. Remote usability testing enables participants to provide feedback on products from their own home, while communicating with researchers via readily available technology, such as video conferencing software. With this method, researchers remotely guide participants through simulated use scenarios for a physical product that has been shipped to their home, and/or for a digital user interface that they can access electronically. You can also use remote interviews to get user feedback on design concepts and potential future design directions. 
 
In addition to usability testing, a formative research technique known as a cognitive walkthrough can be applied to remote research. During a cognitive walkthrough, moderators direct the participant to consider a feature or task, and the participant walks the researcher through what they would do to complete that task. For example, if you wanted to test a graphical user interface that displays patient lab results and gives the option to sort the data in various ways, you can provide participants with a specific task scenario such as, “Imagine that you logged into this system, and you want to sort the lab test results from most recent to oldest.” You might ask participants to think out loud while referring to the interface, to explain how they would navigate, what they would click, and what they would expect to see for results. If users report confusion or describe actions that would take them down an unintended path, designers can use this input to determine optimal design modifications and mitigate any usability issues observed. Although moderators often conduct walkthroughs in-person, with the right audio and video equipment, they can be performed remotely just as well. 
 
Expert and Heuristic Reviews
Without performing any research, human factors and usability experts can provide great design and usability insight by reviewing an interface and identifying potential usability issues based on their knowledge of the device domain, human behavior, understanding of cognition, and ergonomic principles. Once an issue is identified, the usability expert can recommend design considerations that can be implemented in anticipation of a future research effort. Analytical approaches, such as expert and heuristic reviews, are quick and cost effective and can be performed to evaluate physical products, graphical user interfaces, instructions for use (and labeling in general), packaging, and training. Still, it is important to realize that analytical approaches like these don’t replace usability testing, and depending on the extent of the design changes that are derived from analytical approaches, we recommend further evaluating the product to determine whether design mitigations yield the intended result. 
 
Risk Documentation 
Any medical device that needs to claim compliance to IEC62366-1, such as human factors submissions to FDA, requires considerate and deliberate use-related risk documentation, all of which can be created remotely. Documentation might include a Use Failure Modes Effects Analysis (uFMEA) or Use-Related Risk Analysis (URRA). The use-related risk documentation should serve as the basis of all formative and summative research activities, and the basis of these activities is usually a thorough task analysis. A task analysis outlines all of the product’s intended interactions and can be combined with the manufacturer’s clinical risk analysis to create the use-related risk analysis. This allows the medical device manufacturer to define which tasks being performed with the product are considered ‘critical.’ Understanding critical tasks allows designers and human factors researchers to prioritize the most important interactions with the device and understand which aspects should be evaluated in usability studies. 
 
Usability Engineering Documentation
As HF consultants, we often encounter the misconception that the Human Factors Engineering/Usability Engineering (HFE/UE) Summary Report can only be started after a validation study is complete. On the contrary, it’s both efficient and recommended to write most of the HFE/UE report prior to conducting your validation. In the 2016 guidance document, Applying Human Factors and Usability Engineering to Medical Devices, the FDA outlines every section that must be included in the HFE/UE report (from Table A-1 in FDA Guidance document). Sections 2-7 (and even part of section 8) can be written prior to starting the validation study. As a bonus, a good amount of material from the HFE/UE report can be used as part of your validation protocol submission.
 
Sections to be included in HFE/UE Validation Report Submission:
 

• Conclusion
• Description of intended users, uses, use environments, and training
• Description of device user interface
• Summary of known use problems
• Analysis of hazards and risks associated with use of the device
• Summary of preliminary analyses and evaluations
• Description of categorization of critical task
• Details of human factors validation testing

 
Designing a validation study takes a lot of thought and communication between engineers, clinical professionals, designers, and human factors professionals. After a use related risk analysis has been constructed, a validation test protocol can also be created. In short, the background of the protocol defines who the real-life users of the product are, how the product will be used, and where the product will be used. These subjects will provide the material in sections 2 and 3 of the HFE/UE report. When the manufacturer can define these areas, the validation study itself can be constructed.
 
Section 4 of the HFE/UE report is a ‘summary of known use problems’. This section involves researching databases and journals for reports of problems users have had while using similar or predicate devices, such as the MAUDE database. Ideally, this information should be gathered prior to conceptualizing products, as device features and components that have previously caused issues with users should be avoided in future products. When researching databases for information related to novel products, this search may take more time and creativity.
 
Section 8 of the HFE/UE report will present details of the HF validation test, which includes the study results. Identifying the study results obviously requires running the validation study, however, many parts of Section 8 can be combined with the same information written in the validation protocol. For instance, Section 8 will include the rationale for the test type selected, tested environment and conditions of use, number and type of test participants, a description of training, critical tasks required to use the product, etc. All of this will also be included in a validation protocol. If sections 2-7, and part of section 8 is already written, the HFE/UE report becomes much easier to manage after the validation study concludes.
 
Submitting a validation protocol to FDA and waiting for feedback may be another productive activity for your HF program. It’s always recommended to submit an HF Validation protocol to FDA, especially if you have time to submit and wait for the response. Due to the pandemic, manufacturers may be delayed in the ability to recruit participants and hold testing sessions in public spaces, which is why it may be beneficial to submit a protocol while you wait for HF testing to resume. The amount of time it takes to receive a response can vary, but 90 days is a good estimate between submitting the protocol and receiving a response. It is often well worth waiting for the response because it’s possible the FDA may request a drastic study design change that would otherwise be requested upon review of an HFE/UE study report (after a study has been conducted). Requests to change significant aspects surrounding the study design after a study has completed would clearly impact a product timeline for market release and would add significant costs associated with running a supplemental study or another HF validation study.
 
Threshold Analyses 
Developers with Abbreviated New Drug Applications (ANDAs) may choose (or be asked) to prepare a threshold analysis, which does not require in-person usability research. When necessary, it requires internal research teams to compare a new product with a product currently on the market or Reference Listed Drug (RLD). The developer must be able to prove that patients currently using an RLD would not have to experience additional training or have any medical intervention prior to using the new product. The FDA defines three types of threshold analyses when comparing a new product with the RLD: Labeling comparison, comparative task analysis, and physical comparison of the delivery device constituent part. While designing the user interface of the new product, it’s important for the developer to understand the specifics of the RLD user interface so the new product can be designed with the RLD user interface in mind. If the design of the new user interface is driven by the design of the current RLD on the market, then it should be possible to design a comparable product. If manufacturers do not understand the specifics of an RLD user interface prior to designing a product, performing threshold analyses can serve as an activity to understand any differences that may exist. Developers can then use the threshold analysis to redesign any aspects of the new user interface that are found to have external critical design differences. Upon submission to the FDA and review of the threshold analyses, the FDA will make a decision as to whether the new product has critical design differences that may require a user to receive intervention or training prior to using the new product. If the threshold analysis shows signs that the new product is not comparable, developers may need to follow up with a Comparative Use Study.
 
Gap Analyses
If nothing else, now is a great time for development teams to introspect. Gap analyses —wherein existing HF documentation is compared with the list of guidance-suggested submission documents — can be conducted at any point in the human factors development process. They are an excellent means of assessing the completeness of the Design History File, and they provide clarity as to which research and documentation activities still to be conducted before submitting to a regulatory body. 
 
HF Team Training
Manufacturers looking to enter the medical device space may choose to manage some or all HF activities internally. If a consumer technology company begins to develop medical products that require regulatory review, usability experts within the company may not have the expertise or experience to lead a program requiring regulatory procedures. To gain the necessary expertise quickly and effectively, all HF personnel should be trained to navigate medical-specific guidance and comply with regulatory standards. To achieve this, a company may choose to leverage human factors consultants to develop a training program for internal staff who may be responsible for HF activities in the future. To comply with current pandemic safety guidelines and restrictions, this type of training can be conducted via remote streaming. Depending on the types of products the manufacturer develops, the training can be tailored to relevant topics in the industry. Employees can be trained on HF activities that are important throughout an entire product lifecycle, from early generative research to reporting on HF validation activities. Learning national and international human factors engineering guidelines and standards would provide a background for manufacturers to make regulatory decisions, as well as good design decisions independently. In order to conduct HF research independently, employees should be trained on how to moderate usability sessions, perform root cause analysis, and how to effectively communicate research findings to engineers, designers, and regulatory governing bodies.  
 
Conclusion
Despite the challenges of remaining efficient and productive in the face of the pandemic, a lot can be accomplished right now. We still have the ability to adapt and keep product development on track while prioritizing participant, patient, user, and researcher safety. Medical device developers should consider each study, evaluation, and submission as a learning opportunity, ultimately leading to safer and more effective devices. We at Farm, a Flex company, encourage you to employ some of the useful HF methods we have discussed, even those you might not consider under typical circumstances. From competitive landscape initiatives, to remote research studies, to HF team training, now is the time to adapt our human factors activities to create successful medical devices.
 

---

Timothy Visich is an experienced human factors engineer who has planned and executed usability research for a variety of medical related products, and has supported user-centered design and product development for many medical device, pharmaceutical, and consumer technology companies. Visich earned a M.S. degree in Human Factors and Ergonomics from San Jose State University, where he concentrated on product and system improvements in the medical domain.
 
Kate MacNamee is experienced in design research, business strategy, system thinking, and ethnography. Her research specialties include cognitive control and decision making. She earned her MS in cognitive neuroscience – human factors at the University of Utah and received her undergraduate degree in Psych-Neuroscience and German at Colby College in Waterville, Maine.