Maria Shepherd, President and Founder, Medi-Vantage03.04.20
When is the work of human factors over, and how does your team know they are prepared for summative testing? Fact is, human factors is never complete, it is only part of the iterative cycle FDA wants us to use for new product development of medical devices. There is always so much more to learn about your product. Even if the team has completed their analyses on the end-users who will be using the device, assessed the environment(s) where the medical device will be used, and characterized key device user interfaces, your work is still not finished. Is your device ready? Have your design controls prepared your device for summative usability testing?
Why This Is Important
A recent study from Johns Hopkins estimates greater than 250,000 Americans die each year due to medical error, making it the third leading cause of death after heart disease and cancer.1 Other studies state this estimate should be much higher, asserting the number of medical-error-related deaths is as high as 440,000 (Table 1).2 The authors state the discrepancy exists because medical examiners, coroners, physicians, and funeral directors seldom note human errors and system failures on death certificates. Death certificates are the source of information for the CDC when estimating statistics for deaths nationwide. It’s not easy to get this information onto a death certificate.
Table 1: Leading causes of deaths in the U.S. (2017)1-3
Test Early, Test Often
A step many medical device companies skip is formative testing. This is a mistake made for cost control reasons and can have serious implications when it comes time to perform the summative study. Risk errors that may not have been anticipated can and will emerge in the summative (Table 2). Formative usability testing serves to validate the chosen user interface design controls are sufficient to eliminate as many risks to use safety and effectiveness as possible, bringing device performance to an acceptable level. Usability testing enables a medical device design team to improve the usability of their device to meet acceptable standards of risk.
Table 2: Top medical device causes for recalls (Q4, 2018)4
Successful formative testing typically requires five to eight end-users, making the number of iterations flexible for the product development team. Use-related errors can be seen early in the formative usability process. In the early stages of product development, many use-errors can be identified utilizing a methodology that recruits end-users with no prior experience with the device to test how intuitive the device interface is. In the later stages of product development, it is recommended to perform all usability testing with experienced end-users to ensure the product performs well against competitive devices on the market. To control costs, many medtech companies perform early formative testing in house.
A series of formative tests can ensure the end-user will be able to manage use of the device successfully for its intended purpose. When designing formative studies, try to expose as many risks as possible on the road to a successful summative usability test.
What to Avoid
One of the biggest mistakes made in formative usability testing happens when the development team over-trains participants before testing, reducing the potential for use-errors to occur. Also, be sure to capture what end-users find easy and intuitive in order to note design features that do not need to be changed.
The Magic Number
Use-errors are a type of systematic error, so it is difficult to calculate their frequency of occurrence. Table 3 (i.e., K.1 of ISO 62366-2) illustrates the probability of observing at least one instance of a use-error as a function of sample size and underlying population use-error rates.5 It is important to note the underlying population use-error rate can never be known and must be estimated.
Table 3 (i.e., K.1 of ISO 62366-2): Number of test participants needed in a usability test5
ISO 62366-2 states many usability problems can be uncovered with sample sizes from five to eight end-users, driving the industry standard in medical device formative studies.
For summative testing, the ISO standards state 15 end-users per distinct user group (the minimum number recommended by the FDA) is likely to provide the data needed to measure risk. This number is based upon a study conducted in 2003 by Faulkner6 where empirical data was collected from a sample of 60 individuals. The study results indicated a sample of 15 people was enough to find a minimum of 90 percent (average 97 percent) of all problems with the product being tested.
IRB or Not?
When should you perform usability testing under institutional review board (IRB) oversight? An IRB is an organizational body with the goal to protect the rights and welfare of human research subjects participating in research studies. The IRB will want to see the following in your submission:
Purpose of Submitting to IRB
Pre-approval by the IRB is important when potential risks are introduced by the usability study. An obvious example is usability testing of an injection device. This introduces the risk of a needlestick, and an IRB should be utilized to ensure the risk of a needlestick is minimized. Another example is a transdermal patch, where the end-user could have an allergic reaction to patch material. The IRB will ensure the protocol implements protection measures for the end-user participants.7
The Medi-Vantage Perspective
We are frequently asked if there are usability tests that do not require IRB submission. This a decision made on a case-by-case basis and there are reasons to waive an IRB submission, such as if the device is very low risk and the organization managing the usability test has taken all applicable protection measures.
References
Maria Shepherd has more than 20 years of experience in medical device marketing in small startups and top-tier companies. After her industry career, including her role as VP of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing, business strategy, and innovation research for the medical device, diagnostic, and digital health industries. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses and is a member of the Aligo Medtech Investment Committee (www.aligo.com). She can be reached at 855-343-3100. Visit her website at www.medi-vantage.com.
Why This Is Important
A recent study from Johns Hopkins estimates greater than 250,000 Americans die each year due to medical error, making it the third leading cause of death after heart disease and cancer.1 Other studies state this estimate should be much higher, asserting the number of medical-error-related deaths is as high as 440,000 (Table 1).2 The authors state the discrepancy exists because medical examiners, coroners, physicians, and funeral directors seldom note human errors and system failures on death certificates. Death certificates are the source of information for the CDC when estimating statistics for deaths nationwide. It’s not easy to get this information onto a death certificate.
Table 1: Leading causes of deaths in the U.S. (2017)1-3
Test Early, Test Often
A step many medical device companies skip is formative testing. This is a mistake made for cost control reasons and can have serious implications when it comes time to perform the summative study. Risk errors that may not have been anticipated can and will emerge in the summative (Table 2). Formative usability testing serves to validate the chosen user interface design controls are sufficient to eliminate as many risks to use safety and effectiveness as possible, bringing device performance to an acceptable level. Usability testing enables a medical device design team to improve the usability of their device to meet acceptable standards of risk.
Table 2: Top medical device causes for recalls (Q4, 2018)4
Successful formative testing typically requires five to eight end-users, making the number of iterations flexible for the product development team. Use-related errors can be seen early in the formative usability process. In the early stages of product development, many use-errors can be identified utilizing a methodology that recruits end-users with no prior experience with the device to test how intuitive the device interface is. In the later stages of product development, it is recommended to perform all usability testing with experienced end-users to ensure the product performs well against competitive devices on the market. To control costs, many medtech companies perform early formative testing in house.
A series of formative tests can ensure the end-user will be able to manage use of the device successfully for its intended purpose. When designing formative studies, try to expose as many risks as possible on the road to a successful summative usability test.
What to Avoid
One of the biggest mistakes made in formative usability testing happens when the development team over-trains participants before testing, reducing the potential for use-errors to occur. Also, be sure to capture what end-users find easy and intuitive in order to note design features that do not need to be changed.
The Magic Number
Use-errors are a type of systematic error, so it is difficult to calculate their frequency of occurrence. Table 3 (i.e., K.1 of ISO 62366-2) illustrates the probability of observing at least one instance of a use-error as a function of sample size and underlying population use-error rates.5 It is important to note the underlying population use-error rate can never be known and must be estimated.
Table 3 (i.e., K.1 of ISO 62366-2): Number of test participants needed in a usability test5
ISO 62366-2 states many usability problems can be uncovered with sample sizes from five to eight end-users, driving the industry standard in medical device formative studies.
For summative testing, the ISO standards state 15 end-users per distinct user group (the minimum number recommended by the FDA) is likely to provide the data needed to measure risk. This number is based upon a study conducted in 2003 by Faulkner6 where empirical data was collected from a sample of 60 individuals. The study results indicated a sample of 15 people was enough to find a minimum of 90 percent (average 97 percent) of all problems with the product being tested.
IRB or Not?
When should you perform usability testing under institutional review board (IRB) oversight? An IRB is an organizational body with the goal to protect the rights and welfare of human research subjects participating in research studies. The IRB will want to see the following in your submission:
- The usability testing protocol
- Informed consent and assent form
- Recruiting screeners with inclusion and exclusion criteria
- The CV of your designated principal investigator
- Research locations
- Details of your device
Purpose of Submitting to IRB
Pre-approval by the IRB is important when potential risks are introduced by the usability study. An obvious example is usability testing of an injection device. This introduces the risk of a needlestick, and an IRB should be utilized to ensure the risk of a needlestick is minimized. Another example is a transdermal patch, where the end-user could have an allergic reaction to patch material. The IRB will ensure the protocol implements protection measures for the end-user participants.7
The Medi-Vantage Perspective
We are frequently asked if there are usability tests that do not require IRB submission. This a decision made on a case-by-case basis and there are reasons to waive an IRB submission, such as if the device is very low risk and the organization managing the usability test has taken all applicable protection measures.
References
- bit.ly/mpo200301
- bit.ly/mpo200302
- bit.ly/mpo200303
- bit.ly/mpo200304
- bit.ly/mpo200305
- Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, and Computers, 35(3), 379-383.
- bit.ly/mpo200307
Maria Shepherd has more than 20 years of experience in medical device marketing in small startups and top-tier companies. After her industry career, including her role as VP of marketing for Oridion Medical, where she boosted the company valuation prior to its acquisition, director of marketing for Philips Medical, and senior management roles at Boston Scientific Corp., she founded Medi-Vantage. Medi-Vantage provides marketing, business strategy, and innovation research for the medical device, diagnostic, and digital health industries. The firm quantitatively and qualitatively sizes and segments opportunities, evaluates new technologies, provides marketing services, and assesses prospective acquisitions. Shepherd has taught marketing and product development courses and is a member of the Aligo Medtech Investment Committee (www.aligo.com). She can be reached at 855-343-3100. Visit her website at www.medi-vantage.com.
