Rich Newman, Usability Engineering Manager, BlackHägen Design07.30.19
It is becoming common knowledge Human Factors Engineering (HFE), also known as Usability Engineering (UE), is required as part of most medical device development. But what exactly does that entail and how does it affect product development?
A recent Johns Hopkins study suggested medical errors are responsible for over 250,000 deaths each year.1 It has been suggested 15 percent, or almost 40,000, of those errors were related to the device user interface (UI).2 Over the years, the medical device industry has developed methods for ensuring the hardware and software are safe if used as intended. Applying HFE during product development is one method to verify the UI is safe and effective.
The 2016 U.S. Food and Drug Administration (FDA) Human Factors Guidance defines Human Factors Engineering as: “The application of knowledge about human behavior, abilities, limitations, and other characteristics of medical device users to the design of medical devices including mechanical and software driven user interfaces, systems, tasks, user documentation, and user training to enhance and demonstrate safe and effective use.”
A fundamental aspect of HFE is changing how use-related errors are treated. In the past, when a user made a mistake while interacting with a medical device it was called a “User Error” and was considered the user’s fault. However, the mistake likely occurred because the device UI allowed the user to make that mistake, now referred to as a “Use Error.” The FDA HFE guidance defines Use Error as: “User action or lack of action that was different from that expected by the manufacturer and caused a result that (1) was different from the result expected by the user and (2) was not caused solely by device failure and (3) did or could result in harm.”
Common use errors include setting an alarm limit incorrectly, entering an improper dose, misconnecting two components, or the inability to start or stop therapy. A use error that occurs during device use must be treated as a defect like a component failure or software anomaly. While a medical device’s ease of use is still important from a regulatory (validation) perspective, the focus is on Use-Safety, where Use-Safety is defined as freedom from unacceptable use-related risk.
Do I Have to Do HFE?
While HFE has always been encouraged, recent standards and guidance documents drive the requirement to apply HFE. In 2007, an International Standard, IEC 62366, “Application of usability engineering to medical devices,” was released. It was amended in 2012 and revised in 2015. In 2016, the FDA released a guidance document, “Applying Human Factors and Usability Engineering to Medical Devices.”
In the 3rd edition of IEC 60601, part 1-6, the collateral standard for Usability, was revised to reference IEC 62366-1. Manufacturers must show their product meets IEC 62366 to receive CE mark. When submitting to the U.S. FDA, manufacturers must follow the FDA guidance, which is closely aligned with IEC 62366. Both IEC 62366 and the FDA guidance outline a process for developing a safe UI, then validating the final UI is free from unacceptable use-related risk. The process applies to all points of user(s)-product interaction including displays, controls, packaging, product labels, instructions for use (IFU), etc.
The FDA guidance includes criteria for determining when formal UE activities are required. HFE should be applied if the risk analysis result indicates use errors could result in serious harm. HFE processes should also be applied if changes are being made to address a use-related CAPA.
IEC 62366 allows for tailoring, or scaling, the HFE effort based on specific criteria including the UI’s complexity, severity of harm, and the extent of HFE processes applied to an existing medical device. These criteria help determine the amount of generative research and number of UI design iterations and formative evaluations to conduct. Risk analysis and summative validation are not subject to tailoring.
The HFE Process: The HFE process is risk-based, similar to those used in other aspects of development. The process involves identifying potential use-related hazards, mitigating those risks, and validating the risk mitigations are effective.
Pre-Design Controls HFE Activities: To develop a safe and effective UI, first understand how the device is used, who uses it, which use environment aspects could impact its use, etc. This information is obtained by conducting generative research such as Contextual Inquiry (observing device use in context), mining complaint databases, and conducting expert reviews of existing devices. Output of these activities is documented in the Use Specification.
Use-Related Risk Assessment: Perform the Use-Related Risk Assessment (URRA) to identify known or foreseeable hazards and identify hazard-related use scenarios. A URRA can use methods derived from analysis tools such as FMEA as well as HFE-specific methods and is part of the system risk analysis. Because probability is very difficult to determine for use errors and many use errors cannot be anticipated until device use is simulated and observed, URRA focuses on potential harm severity rather than probability.3 Probability of occurrence is generally not included in a URRA. The URRA is used to inform the UI design and determine what tasks and use scenarios to include in formative evaluations and the Summative Usability Validation.
UI Design and Evaluation: UI design is an iterative process. Each iteration consists of UI design, prototyping or implementation, and evaluation. Each iteration builds on the learnings from previous cycles. Early iterations might employ sketches, static prototypes, or GUI simulations running on a laptop or tablet. Later iterations typically use working prototypes, software, and eventually pre-production devices. The fidelity of the prototypes (and the usability evaluation) and amount of functionality evaluated continue to increase as the product is developed. The number of iterations varies.
Usability Testing Overview: Usability testing is one of the most common and effective methods of evaluating a device UI. Usability tests are typically conducted as simulated use studies, where representative users interact with prototypes or actual devices and perform “hands on” tasks that simulate the actual tasks performed during use. Testing is conducted in an environment that simulates aspects of environment that might impact the ability to safely use the device, such as noise, lighting, or stress. UI aspects not easily or safely simulated are evaluated as knowledge-based tasks, where users are asked questions to assess their understanding of that UI aspect.
Testing is generally performed in a usability suite with an attached observation room that allows observers to view testing through a two-way mirror without influencing the test. If a higher fidelity environment is required, testing can be conducted in a medical simulation center or similar facility.
Formative Evaluations: Evaluations conducted during UI design are referred to as formative evaluations. The purpose of Formative Evaluations is to inform the UI design. The focus is on finding the root cause of potential use errors with the intent of understanding how to prevent them and mitigate the risk. They may also identify unanticipated use errors or hazards and inform the design of training, labeling, and the summative test. While not the primary focus, formative evaluations may also yield insights on usability issues that can be used to improve ease-of-use for the product. Formative evaluation methods include expert reviews, user preference tests, cognitive walkthroughs (guided interviews), and formative usability tests.
Summative Validation Testing: A Summative Validation test is a more formal usability test intended to validate the device’s use-safety. Devices used for summative testing must be production equivalent. If labeling, IFU, and/or training are mitigations for use-related risk, they must also be validated. On the other hand, training should not be included if the user might not be expected to receive formal training prior to using the device. If summative participants are trained, there should be an appropriate break between training and testing to simulate the “falling-off of knowledge” that would occur in the time between training and when the user would use the device for the first time. Data is gathered during summative testing to identify the root cause of use errors and difficulties with the intent of informing a residual risk analysis.
Residual Risk Analysis: The final step is to perform a residual risk analysis. It is expected there will be use errors and difficulties observed during the summative test. The results of the summative test should be analyzed to show the UI is safe and effective and the use-related risk has been reduced as much as is reasonably practicable.
Integrating UE into the PDP
Including HFE activities in a product development plan (PDP) can help reduce the impact of those activities on the project schedule and cost. Points to consider include:
References
Rich Newman is the Usability Engineering Manager at BlackHägen Design, an R&D and HFE consultancy. Rich has over 25 years of experience in the medical device industry, starting as a software engineer before transitioning to Human Factors. His prior experience includes ZOLL Medical, Philips Healthcare, and DEKA Research.
A recent Johns Hopkins study suggested medical errors are responsible for over 250,000 deaths each year.1 It has been suggested 15 percent, or almost 40,000, of those errors were related to the device user interface (UI).2 Over the years, the medical device industry has developed methods for ensuring the hardware and software are safe if used as intended. Applying HFE during product development is one method to verify the UI is safe and effective.
The 2016 U.S. Food and Drug Administration (FDA) Human Factors Guidance defines Human Factors Engineering as: “The application of knowledge about human behavior, abilities, limitations, and other characteristics of medical device users to the design of medical devices including mechanical and software driven user interfaces, systems, tasks, user documentation, and user training to enhance and demonstrate safe and effective use.”
A fundamental aspect of HFE is changing how use-related errors are treated. In the past, when a user made a mistake while interacting with a medical device it was called a “User Error” and was considered the user’s fault. However, the mistake likely occurred because the device UI allowed the user to make that mistake, now referred to as a “Use Error.” The FDA HFE guidance defines Use Error as: “User action or lack of action that was different from that expected by the manufacturer and caused a result that (1) was different from the result expected by the user and (2) was not caused solely by device failure and (3) did or could result in harm.”
Common use errors include setting an alarm limit incorrectly, entering an improper dose, misconnecting two components, or the inability to start or stop therapy. A use error that occurs during device use must be treated as a defect like a component failure or software anomaly. While a medical device’s ease of use is still important from a regulatory (validation) perspective, the focus is on Use-Safety, where Use-Safety is defined as freedom from unacceptable use-related risk.
Do I Have to Do HFE?
While HFE has always been encouraged, recent standards and guidance documents drive the requirement to apply HFE. In 2007, an International Standard, IEC 62366, “Application of usability engineering to medical devices,” was released. It was amended in 2012 and revised in 2015. In 2016, the FDA released a guidance document, “Applying Human Factors and Usability Engineering to Medical Devices.”
In the 3rd edition of IEC 60601, part 1-6, the collateral standard for Usability, was revised to reference IEC 62366-1. Manufacturers must show their product meets IEC 62366 to receive CE mark. When submitting to the U.S. FDA, manufacturers must follow the FDA guidance, which is closely aligned with IEC 62366. Both IEC 62366 and the FDA guidance outline a process for developing a safe UI, then validating the final UI is free from unacceptable use-related risk. The process applies to all points of user(s)-product interaction including displays, controls, packaging, product labels, instructions for use (IFU), etc.
The FDA guidance includes criteria for determining when formal UE activities are required. HFE should be applied if the risk analysis result indicates use errors could result in serious harm. HFE processes should also be applied if changes are being made to address a use-related CAPA.
IEC 62366 allows for tailoring, or scaling, the HFE effort based on specific criteria including the UI’s complexity, severity of harm, and the extent of HFE processes applied to an existing medical device. These criteria help determine the amount of generative research and number of UI design iterations and formative evaluations to conduct. Risk analysis and summative validation are not subject to tailoring.
The HFE Process: The HFE process is risk-based, similar to those used in other aspects of development. The process involves identifying potential use-related hazards, mitigating those risks, and validating the risk mitigations are effective.
Pre-Design Controls HFE Activities: To develop a safe and effective UI, first understand how the device is used, who uses it, which use environment aspects could impact its use, etc. This information is obtained by conducting generative research such as Contextual Inquiry (observing device use in context), mining complaint databases, and conducting expert reviews of existing devices. Output of these activities is documented in the Use Specification.
Use-Related Risk Assessment: Perform the Use-Related Risk Assessment (URRA) to identify known or foreseeable hazards and identify hazard-related use scenarios. A URRA can use methods derived from analysis tools such as FMEA as well as HFE-specific methods and is part of the system risk analysis. Because probability is very difficult to determine for use errors and many use errors cannot be anticipated until device use is simulated and observed, URRA focuses on potential harm severity rather than probability.3 Probability of occurrence is generally not included in a URRA. The URRA is used to inform the UI design and determine what tasks and use scenarios to include in formative evaluations and the Summative Usability Validation.
UI Design and Evaluation: UI design is an iterative process. Each iteration consists of UI design, prototyping or implementation, and evaluation. Each iteration builds on the learnings from previous cycles. Early iterations might employ sketches, static prototypes, or GUI simulations running on a laptop or tablet. Later iterations typically use working prototypes, software, and eventually pre-production devices. The fidelity of the prototypes (and the usability evaluation) and amount of functionality evaluated continue to increase as the product is developed. The number of iterations varies.
Usability Testing Overview: Usability testing is one of the most common and effective methods of evaluating a device UI. Usability tests are typically conducted as simulated use studies, where representative users interact with prototypes or actual devices and perform “hands on” tasks that simulate the actual tasks performed during use. Testing is conducted in an environment that simulates aspects of environment that might impact the ability to safely use the device, such as noise, lighting, or stress. UI aspects not easily or safely simulated are evaluated as knowledge-based tasks, where users are asked questions to assess their understanding of that UI aspect.
Testing is generally performed in a usability suite with an attached observation room that allows observers to view testing through a two-way mirror without influencing the test. If a higher fidelity environment is required, testing can be conducted in a medical simulation center or similar facility.
Formative Evaluations: Evaluations conducted during UI design are referred to as formative evaluations. The purpose of Formative Evaluations is to inform the UI design. The focus is on finding the root cause of potential use errors with the intent of understanding how to prevent them and mitigate the risk. They may also identify unanticipated use errors or hazards and inform the design of training, labeling, and the summative test. While not the primary focus, formative evaluations may also yield insights on usability issues that can be used to improve ease-of-use for the product. Formative evaluation methods include expert reviews, user preference tests, cognitive walkthroughs (guided interviews), and formative usability tests.
Summative Validation Testing: A Summative Validation test is a more formal usability test intended to validate the device’s use-safety. Devices used for summative testing must be production equivalent. If labeling, IFU, and/or training are mitigations for use-related risk, they must also be validated. On the other hand, training should not be included if the user might not be expected to receive formal training prior to using the device. If summative participants are trained, there should be an appropriate break between training and testing to simulate the “falling-off of knowledge” that would occur in the time between training and when the user would use the device for the first time. Data is gathered during summative testing to identify the root cause of use errors and difficulties with the intent of informing a residual risk analysis.
Residual Risk Analysis: The final step is to perform a residual risk analysis. It is expected there will be use errors and difficulties observed during the summative test. The results of the summative test should be analyzed to show the UI is safe and effective and the use-related risk has been reduced as much as is reasonably practicable.
Integrating UE into the PDP
Including HFE activities in a product development plan (PDP) can help reduce the impact of those activities on the project schedule and cost. Points to consider include:
- Align UI iterations with major development milestones. Availability of prototypes drives the usability testing schedule, not vice-versa.
- Plan for the effort to create the stimuli needed for usability testing, whether dedicated prototypes, “hardened” software versions, etc.
- Include time in the schedule to make UI changes in response to formative test findings. Although one doesn’t know what will be found, expect to find at least one issue requiring significant changes, especially during early formatives.
- Be conservative about schedule estimates. Rescheduling usability tests because prototypes were late can be very costly, especially after participant recruiting starts.
- If training or accompanying documentation (IFU) is used to mitigate risk, they should be evaluated in formatives prior to validation.
References
- http://bit.ly/mpo190761
- Estimate by former FDA Human Factors Chief Peter Carstensen in mid 1990s
- FDA Human Factors Guidance, 2016, pg 6
Rich Newman is the Usability Engineering Manager at BlackHägen Design, an R&D and HFE consultancy. Rich has over 25 years of experience in the medical device industry, starting as a software engineer before transitioning to Human Factors. His prior experience includes ZOLL Medical, Philips Healthcare, and DEKA Research.
