Bruce Johnston, Senior Software Engineering Specialist, MedAcuity Software04.01.20
Safety is one of the most important issues to consider for medical device software development. The user experience with such devices plays a significant role in determining whether they can be used correctly and safely both today and in the future. Adding the consideration of the usability of a product and the risks its use may pose during medical software development ultimately enables OEMs to offer safe, innovative new devices that will consistently perform as intended.
Software has the power to improve a patient’s quality of life and drive better outcomes. But design problems with medical software products and devices contribute to thousands of injuries and deaths each year. Software design failures stemming from inadequate development practices, poor designs, coding errors, and cybersecurity vulnerabilities are the leading cause of medical device recalls. Many of these problems are set in place before the first line of code is even written and could be prevented by the application of well-known software development best practices.
Human Factors and Usability for Medical Software
The regulation of medical software can be used as a means to get software developers to think about their project and how that project is going to be put together to achieve its goals through the application of software development best practices.
One example of this is the guidance from the FDA concerning the usability of medical software. Software development tends to focus on what the software should do. How an operator will get the software to do what it should do is often a secondary consideration. With this guidance, the FDA is asking software developers to think about how their software will be used by people for which it is intended. Under this guidance, mistakes made by operators using a software (use errors) are considered to be a problem with the software, not with the user.
Focus on Usability
Despite the use of development best practices, under intense regulatory scrutiny, a significant number of medical device incidents and recalls involve a use error or other user interface issues. It’s the number of such incidents and recalls that has drawn attention to the usability of medical devices and software.
Part of the problem may be that it is easy to forget how hard something is to do when you do it all day, every day. Software development teams become so familiar with their products, they don’t see the difficulties someone else may have interacting with it. Assumptions are often made, either explicitly or implicitly, about the expertise and training of the users of the software or device. Software developers may imagine well-trained operators who have memorized the user’s manual standing in front of their devices carefully entering information and reading every prompt and bit of text displayed. Reality is often very different. The device may be one of several dozen different devices from different manufacturers on a floor, half of which are beeping or have screeching alarms. An understaffed, overworked team that may be months behind on its training is dashing from device to device and has perhaps 10 to 15 seconds to interact with the device before something else demands their attention. These scenarios represent two extremes; the real world lies somewhere in between, but it is likely closer to the latter.
The FDA is seeking to focus attention on these concerns with its recent guidance on usability and human factors analysis in the development of medical software. All too often “we’ll address that in the user’s manual” has become the work-around for potential usability problems. Issues seen in the field are often explained away as a mistake made by an incompetent or poorly trained user. With this latest guidance, however, use errors made by operators of a software device are considered to be a problem with the software, not with the user. The new paradigm is to design in usability, and design out use error.
The main tool to assess the usability of a software device and drive the process of “designing out use error” is the human factors analysis. Manufacturers seeking FDA approval for new devices must submit evidence of systematic human factors analysis of use errors.
Human Factors Analysis: What Does It Mean?
Human factors analysis, in the context of medical software development, has three main goals:
The human factors analysis ties into several aspects of the development process. It informs important design decisions early in the process, but also lays the foundation for usability acceptance testing that will validate the resulting software meets its design goals.
A key aspect of human factors analysis is the analysis process must begin early enough to identify issues and incorporate remedies into the design. Waiting until post-implementation testing to identify usability issues will mean the remedies will be much costlier to implement because the relatively easy, late-stage work-arounds currently used that involve instructions for use, warnings, and labelling are no longer considered sufficient to address risks posed by usability issues.
There are several complementary analysis types that, when performed over the lifecycle of the software, together constitute what the FDA refers to as “systematic” human factors analysis. They include use error and task analysis, early prototype usability evaluation, usability testing, and post-market usability studies. Each of these types of analyses are necessary but, by themselves, are not sufficient to ensure the usability of the software. Together, however, they form the foundation for the development of a safe and usable system.
Use error and task analysis is the first step in the process of identifying possible sources of use errors. Very early in the development process, before software designs are created and certainly before any code is written, the design team works through paper and mental exercises with the goal of understanding how, by whom, and in what environment the software will be used. The experience gained by such exercises is essential to give the designers a basic understanding of the software from the perspective of the people who will use it.
Getting the design team to think about the user perspective is necessary if they are to create usable software, but that alone is not sufficient. There is no substitute for actual feedback on the software from real users. To get such feedback early requires the second type of human factors analysis: early prototype usability evaluation. Early prototyping is an essential source of input on the potential usability of the software from people representative of the targeted user community. Creating a safe, adoptable, and user-friendly medical device experience should incorporate feedback reflective of the entire user ecosystem, including patients, doctors, nurses, technicians, and maintenance providers. The main goal of such evaluation is to get users to identify bottlenecks in workflow and possible misunderstandings of instructions or displayed information.
Walk-through-talk-through demonstrations and hands-on sessions are an effective means to solicit feedback on displays, readability, sounds, and basic flow. The value of such feedback is, however, directly related to how “real” the prototype is. While value can be obtained through storyboards and wireframe mockups, there is no substitute for seeing how an operator interacts with a functioning UI.
As a design matures and implementation gets underway, formal usability testing can start to provide feedback on not just user interface interactions, but also on the user’s ability to execute tasks and perform required functions in near-real world environments. Formal usability testing continues through the development process and evolves into an important aspect of both the verification of the software and validation of the completed product.
Usability concerns do not, however, end with the validation and deployment of a product to the field. Because it is difficult to simulate all real-world scenarios for use of a product, post-market usability studies or surveys need to continue after the initial release in order to reveal any needs that were not met or could not have been anticipated. Customer service reports are a rich source of information on issues users are having with the software and should be reviewed to identify patterns and “hot-spots” that might indicate usability or UI design issues. Post-market feedback is essential to drive both design improvements and development process improvements for future releases and products.
The Right Tools for the Job
A hammer is a great tool. It’s well designed with heft where it is needed to drive the task home, but with balance to avoid fatiguing the user. Many even have a built-in claw to help correct mistakes. A hammer is a great tool, perfectly designed to meet the needs of the task at hand...unless that task is to cut 4 inches off the end of a 2x4. Every software developer has experienced the feeling of fighting with a tool that felt as well suited to its task as a hammer to cutting wood. Having a good tool is necessary, but not sufficient; the tool must be the right tool for the task at hand.
Having the right tools for usability studies and human factors analysis is an important factor in their success. There are many tools available to help build user interfaces. Selection of a UI tool will be driven by a number of factors including the target environment, performance requirements, platform constraints, cost, and other elements of the development tool chain with which the UI tools must work (IDEs, languages, etc.). Added to this list of drivers is the ability to effectively support human factors analysis and usability testing.
The need for the UI tools to support usability testing is most critical in the early stages of the design process where quick prototype development and modification is essential to provide working user interface prototypes that can be put in front of representative users. There are a number of good tools available to provide quick UI mockups and prototypes. It is important to remember when looking at potential tools that human factors analysis doesn’t end after the initial round of usability testing.
To be most effective, human factors analysis must be integrated into the development process. When human factors analysis is done in isolation, much can get “lost in translation” when the software team implements the actual user interface. The best tools will produce early prototypes of the actual product user interface that will evolve with the rest of the development effort into stable designs and, eventually, into the finished product. A “throw-away” prototype is a waste and the cost of producing and maintaining them will pressure teams into less frequent usability testing and human factors analysis. When early prototypes are early implementations, the result is better feedback from testing (users are using real software) and more efficient development (there is no translation, reimplementation, or lost effort).
Another important aspect of the user interface tools is the ability to separate the presentation of task and workflow in the UI from the back-end code, logic, and algorithms. This separation facilitates updates and changes to the user interface throughout the design and implementation process. Usability issues (identified through the ongoing process of human factors analysis) can be corrected in the user interface with minimal disruption in the back-end implementations, even late in the development process.
Conclusion
The goal of software development teams is to build great products and minimize the risk of failure. This is most often achieved through the application of industry best practices developed from years of software development experience. Medical software development requires these best practices and adds the consideration of the safety of the product and the risks it may pose to its users.
Regulations have been put in place to ensure compliance with already-established best practices and prompt developers to think beyond their local environments to consider the implications of the deployment of their products into real-world scenarios. When it comes to usability, the focus needs to shift away from what a developer wants a technology to do and toward what the user and patient need it to do.
The right tools are essential to help drive this process and facilitate early and ongoing interaction of the software with users to gain insight and identify problems. Seamless evolution of early prototypes into production software allows development teams to efficiently and effectively build, demonstrate, test, vet, and adapt designs to match the needs, capabilities, and limitations of those users. This is an essential part of the process of developing a safe and effective software for medical applications.
Dr. Bruce Johnston of MedAcuity Software has more than 25 years of experience designing and developing complex software systems and more than 15 years of experience working in regulated medical software development. Much of this experience has be focused on UI design and human factors aspects of Class II and Class III medical device programs. Most recently, Dr. Johnston has been providing strategic insight to a variety of clients on the intersection of modern software development methodologies, including HF/Usability and Agile practices, and the regulated medical environment.
Software has the power to improve a patient’s quality of life and drive better outcomes. But design problems with medical software products and devices contribute to thousands of injuries and deaths each year. Software design failures stemming from inadequate development practices, poor designs, coding errors, and cybersecurity vulnerabilities are the leading cause of medical device recalls. Many of these problems are set in place before the first line of code is even written and could be prevented by the application of well-known software development best practices.
Human Factors and Usability for Medical Software
The regulation of medical software can be used as a means to get software developers to think about their project and how that project is going to be put together to achieve its goals through the application of software development best practices.
One example of this is the guidance from the FDA concerning the usability of medical software. Software development tends to focus on what the software should do. How an operator will get the software to do what it should do is often a secondary consideration. With this guidance, the FDA is asking software developers to think about how their software will be used by people for which it is intended. Under this guidance, mistakes made by operators using a software (use errors) are considered to be a problem with the software, not with the user.
Focus on Usability
Despite the use of development best practices, under intense regulatory scrutiny, a significant number of medical device incidents and recalls involve a use error or other user interface issues. It’s the number of such incidents and recalls that has drawn attention to the usability of medical devices and software.
Part of the problem may be that it is easy to forget how hard something is to do when you do it all day, every day. Software development teams become so familiar with their products, they don’t see the difficulties someone else may have interacting with it. Assumptions are often made, either explicitly or implicitly, about the expertise and training of the users of the software or device. Software developers may imagine well-trained operators who have memorized the user’s manual standing in front of their devices carefully entering information and reading every prompt and bit of text displayed. Reality is often very different. The device may be one of several dozen different devices from different manufacturers on a floor, half of which are beeping or have screeching alarms. An understaffed, overworked team that may be months behind on its training is dashing from device to device and has perhaps 10 to 15 seconds to interact with the device before something else demands their attention. These scenarios represent two extremes; the real world lies somewhere in between, but it is likely closer to the latter.
The FDA is seeking to focus attention on these concerns with its recent guidance on usability and human factors analysis in the development of medical software. All too often “we’ll address that in the user’s manual” has become the work-around for potential usability problems. Issues seen in the field are often explained away as a mistake made by an incompetent or poorly trained user. With this latest guidance, however, use errors made by operators of a software device are considered to be a problem with the software, not with the user. The new paradigm is to design in usability, and design out use error.
The main tool to assess the usability of a software device and drive the process of “designing out use error” is the human factors analysis. Manufacturers seeking FDA approval for new devices must submit evidence of systematic human factors analysis of use errors.
Human Factors Analysis: What Does It Mean?
Human factors analysis, in the context of medical software development, has three main goals:
- To gain understanding of how the software/device will be used, including who will be using it and in what environment.
- To drive a design that matches the capabilities and limitations of those human users of the software or device.
- To discover possible paths to use errors and eliminate them from the design.
The human factors analysis ties into several aspects of the development process. It informs important design decisions early in the process, but also lays the foundation for usability acceptance testing that will validate the resulting software meets its design goals.
A key aspect of human factors analysis is the analysis process must begin early enough to identify issues and incorporate remedies into the design. Waiting until post-implementation testing to identify usability issues will mean the remedies will be much costlier to implement because the relatively easy, late-stage work-arounds currently used that involve instructions for use, warnings, and labelling are no longer considered sufficient to address risks posed by usability issues.
There are several complementary analysis types that, when performed over the lifecycle of the software, together constitute what the FDA refers to as “systematic” human factors analysis. They include use error and task analysis, early prototype usability evaluation, usability testing, and post-market usability studies. Each of these types of analyses are necessary but, by themselves, are not sufficient to ensure the usability of the software. Together, however, they form the foundation for the development of a safe and usable system.
Use error and task analysis is the first step in the process of identifying possible sources of use errors. Very early in the development process, before software designs are created and certainly before any code is written, the design team works through paper and mental exercises with the goal of understanding how, by whom, and in what environment the software will be used. The experience gained by such exercises is essential to give the designers a basic understanding of the software from the perspective of the people who will use it.
Getting the design team to think about the user perspective is necessary if they are to create usable software, but that alone is not sufficient. There is no substitute for actual feedback on the software from real users. To get such feedback early requires the second type of human factors analysis: early prototype usability evaluation. Early prototyping is an essential source of input on the potential usability of the software from people representative of the targeted user community. Creating a safe, adoptable, and user-friendly medical device experience should incorporate feedback reflective of the entire user ecosystem, including patients, doctors, nurses, technicians, and maintenance providers. The main goal of such evaluation is to get users to identify bottlenecks in workflow and possible misunderstandings of instructions or displayed information.
Walk-through-talk-through demonstrations and hands-on sessions are an effective means to solicit feedback on displays, readability, sounds, and basic flow. The value of such feedback is, however, directly related to how “real” the prototype is. While value can be obtained through storyboards and wireframe mockups, there is no substitute for seeing how an operator interacts with a functioning UI.
As a design matures and implementation gets underway, formal usability testing can start to provide feedback on not just user interface interactions, but also on the user’s ability to execute tasks and perform required functions in near-real world environments. Formal usability testing continues through the development process and evolves into an important aspect of both the verification of the software and validation of the completed product.
Usability concerns do not, however, end with the validation and deployment of a product to the field. Because it is difficult to simulate all real-world scenarios for use of a product, post-market usability studies or surveys need to continue after the initial release in order to reveal any needs that were not met or could not have been anticipated. Customer service reports are a rich source of information on issues users are having with the software and should be reviewed to identify patterns and “hot-spots” that might indicate usability or UI design issues. Post-market feedback is essential to drive both design improvements and development process improvements for future releases and products.
The Right Tools for the Job
A hammer is a great tool. It’s well designed with heft where it is needed to drive the task home, but with balance to avoid fatiguing the user. Many even have a built-in claw to help correct mistakes. A hammer is a great tool, perfectly designed to meet the needs of the task at hand...unless that task is to cut 4 inches off the end of a 2x4. Every software developer has experienced the feeling of fighting with a tool that felt as well suited to its task as a hammer to cutting wood. Having a good tool is necessary, but not sufficient; the tool must be the right tool for the task at hand.
Having the right tools for usability studies and human factors analysis is an important factor in their success. There are many tools available to help build user interfaces. Selection of a UI tool will be driven by a number of factors including the target environment, performance requirements, platform constraints, cost, and other elements of the development tool chain with which the UI tools must work (IDEs, languages, etc.). Added to this list of drivers is the ability to effectively support human factors analysis and usability testing.
The need for the UI tools to support usability testing is most critical in the early stages of the design process where quick prototype development and modification is essential to provide working user interface prototypes that can be put in front of representative users. There are a number of good tools available to provide quick UI mockups and prototypes. It is important to remember when looking at potential tools that human factors analysis doesn’t end after the initial round of usability testing.
To be most effective, human factors analysis must be integrated into the development process. When human factors analysis is done in isolation, much can get “lost in translation” when the software team implements the actual user interface. The best tools will produce early prototypes of the actual product user interface that will evolve with the rest of the development effort into stable designs and, eventually, into the finished product. A “throw-away” prototype is a waste and the cost of producing and maintaining them will pressure teams into less frequent usability testing and human factors analysis. When early prototypes are early implementations, the result is better feedback from testing (users are using real software) and more efficient development (there is no translation, reimplementation, or lost effort).
Another important aspect of the user interface tools is the ability to separate the presentation of task and workflow in the UI from the back-end code, logic, and algorithms. This separation facilitates updates and changes to the user interface throughout the design and implementation process. Usability issues (identified through the ongoing process of human factors analysis) can be corrected in the user interface with minimal disruption in the back-end implementations, even late in the development process.
Conclusion
The goal of software development teams is to build great products and minimize the risk of failure. This is most often achieved through the application of industry best practices developed from years of software development experience. Medical software development requires these best practices and adds the consideration of the safety of the product and the risks it may pose to its users.
Regulations have been put in place to ensure compliance with already-established best practices and prompt developers to think beyond their local environments to consider the implications of the deployment of their products into real-world scenarios. When it comes to usability, the focus needs to shift away from what a developer wants a technology to do and toward what the user and patient need it to do.
The right tools are essential to help drive this process and facilitate early and ongoing interaction of the software with users to gain insight and identify problems. Seamless evolution of early prototypes into production software allows development teams to efficiently and effectively build, demonstrate, test, vet, and adapt designs to match the needs, capabilities, and limitations of those users. This is an essential part of the process of developing a safe and effective software for medical applications.
Dr. Bruce Johnston of MedAcuity Software has more than 25 years of experience designing and developing complex software systems and more than 15 years of experience working in regulated medical software development. Much of this experience has be focused on UI design and human factors aspects of Class II and Class III medical device programs. Most recently, Dr. Johnston has been providing strategic insight to a variety of clients on the intersection of modern software development methodologies, including HF/Usability and Agile practices, and the regulated medical environment.
