Three Crucial Elements of Conducting International Usability Research

It is important to conduct usability research throughout the product development cycle. In the United States, this process is advocated and regulated by the U.S. Food and Drug Administration (FDA). Internationally, regulations vary but typically follow U.S. guidance and those found in IEC 62366-1 and -2. (The scope of this article focuses on evaluative user research and does not include generative methods such as contextual inquiry.) Early in the process, there is a need to confirm confidence in the design and inform system architecture decisions via user preference testing. Once the design is defined, formative studies focusing on user interface details may be conducted. These studies allow for the discovery of new risks while ensuring the design employs proper risk mitigations.

For a U.S.-based device, this process is straightforward. However, what if you are designing a device seeking initial clearance in the U.S., but will eventually transition to international markets? What if the device will be released in OUS (outside U.S.) markets then be adapted back to the U.S., or be an OUS offering exclusively? It is usually necessary to conduct usability studies overseas to receive regulatory approval for your device. Language barriers, cultural subtleties, long-distance logistics, and various unknowns can easily result in disastrous outcomes. The following best practices are wisdom from the field and should ensure valuable data is collected for your project.

1. Logistics
The first challenge—possibly one of the greatest a project team must overcome for usability research—is logistics. Moving prototypes around the globe, arranging study facilities, recruiting, and ensuring competent partners are prepared can be daunting.

One of the critical issues R&D teams should be concerned with is packaging and shipping prototypes. International shipping is entirely different than domestic logistics and success is paramount to ensure prototypes arrive in time for research activities. When shipping prototypes, ensure nothing is labeled as a “medical” or “clinical” device and everything is clearly labeled “Not for Human Use.” Regardless of where prototypes are shipped, customs will identify a medical device as a high-risk item and investigate to ensure proper taxes and tariffs have been satisfied before releasing the shipment. This will likely result in significant delays and days, if not weeks, of communication to release prototypes. Check local regulations or work with a local expediter to ensure the shipment is able to clear customs without delays. Take the same precautions when returning prototypes to the U.S., as imports can be just as challenging.

Packaging should also be a strong concern for prototypes while shipping internationally. Obviously, package the prototypes in a robust manner to prevent damage in transit; however, many people fail to remember numerous international shipments are opened and inspected prior to delivery. When packaging prototypes, make sure the customs paperwork and packing list is easily accessible if the package is opened. Enable the package to be easily re-packed and be conscientious of how you describe the contents on the manifest. It is always best practice to carry the most critical components with you in your luggage for the best chance of success. If possible, create a virtual prototype of the user interface so the research activity can still be salvaged—even if the study stimuli don’t arrive on time. Check with the shipping provider and discuss the logistics early in the planning for research activities to ensure an appropriate amount of time has been estimated for project planning. Plan for setting up and testing everything with a day of buffer to act as a contingency if possible.

While conducting usability research activities internationally, it is also important to maximize the return on investment and collect as much data as possible. So much time and resources will have been invested that you cannot redo any portion of the study, so too much data is better than missing data. There will be multiple actors for lab-based studies, each creating their own data. When possible, always record the participant’s actions via multiple video cameras and the audio feeds from the room, the native-speaking moderator, and the simultaneous interpreter. Each of these audio and video feeds are recorded individually, then later mixed together. This allows the analysis team to hone in on each individual audio track. Additionally, a native language video can be provided back to the moderator or client for further analysis. All these individual data feeds produce a significant amount of insights. Plan ahead and bring appropriate technology to back up the data before traveling back to the states. Do not underestimate the amount of data that will be created; one of our recent one-week studies produced over 1.5 TB of video and audio data. Additionally, check local privacy and confidentiality laws. The EU is currently stricter than the United States and this requires different strategies for collecting and managing data.

2. Language Barrier
Selecting an appropriate recruitment partner is one of the most crucial elements of a successful international usability study. The recruitment partner should have experience conducting usability studies and medical device research. It can be beneficial to locate a recruitment partner who also has experience conducting pharmaceutical research, as they may be privy to local laws and regulations pertaining to clinician compensation. Having a base level of medical knowledge is important to ensure the recruitment partner can access local clinicians for your study. The recruitment partner should have the ability to arrange on-site logistics and have a shipping address for incoming prototypes. At the end of the day, the recruitment partner will be the best resource for local knowledge. They will have the ability to temper expectations and inform you of local culture and practices. Feel comfortable interviewing multiple recruitment partners to ensure they are a good fit for your company and project.

The recruitment partner should have the ability to recommend staffing resources for your study. At a minimum, you will need a simultaneous interpreter and a native-speaking moderator. Even if the participants speak English, it is best to bring on a local moderator because they are familiar with local cultural norms and non-verbal cues. For example, in certain cultures it is best to run through a study as fast as possible with the least formalities. However, in some Asian countries, it is recommended to allow time for a break in the middle of a study to create a formal environment for the participants.

While it may seem inconsequential, it is best to find a moderator and interpreter that have collaborated before. The moderator will be the lead to your study and the direct interface to your participants. The simultaneous interpreter will be the bridge between yourself and the study’s activities. If the interpreter and moderator have worked together before, they will be cognizant of each other’s pace and cadence throughout the study’s activities. Much like the recruitment agency, it is important to review this pair’s experience to ensure they have sufficient knowledge of the medical ecosystem. Once the study staff that will partner with your company has been identified, plan on having multiple collaboration meetings to review the study protocol during its development. After arriving in the study locale, meet with the study staff to review the final protocol and run through a rehearsal to make sure everyone’s expectations have been met.

Typically, study stimuli will require input from the study facilitators to offer the most immersive experience for the participants. An engineer may need to operate prototype features that may not be entirely functional at the time of the study. Study facilitators must create a system of communication between the moderator and themselves to operate the prototype or study stimuli in sync with the study; language barriers may make this difficult. Radios and hands-free ear pieces can be used for the simultaneous interpreter to communicate with the study facilitators discreetly and efficiently.

3. Stakeholders
It requires multiple individuals to pull off a successful international study. Planning, logistics, study execution, and data analysis all require their own set of expertise.

A protocol author is an obvious requirement, and no different than what’s required for a domestic study. However, the author must have the ability to communicate and collaborate with the international resources. During the early planning stages, the protocol author should lead the charge in developing and planning the study activities. Whoever is filling this role will be the most familiar with the study tasks and should be very intimate with the study stimuli. Multiple collaboration meetings between the author and R&D team are necessary to ensure the author is familiar with the prototypes or study stimuli. The author will also be the main point of contact between the simultaneous interpreter, moderator, and study site. They should have a flexible schedule to ensure collaboration can occur over multiple time zones.

As previously discussed, a successful study requires a simultaneous interpreter and native-speaking moderator. The day of activities will be handled by a set of study facilitators. It is best to have a usability testing specialist and an R&D engineer to staff the study. This combination is important, as the usability specialist will have the skills and knowledge to alter the study protocol or handle any issues that may occur with the study tasks. The R&D engineer is particularly important if the prototypes require emergency fixes. Understandably, it is important both have extensive knowledge of the study tasks and the study stimuli.

While international usability research may seem daunting—especially in comparison to domestic research—it is the best way to ensure that you gather the correct data for the intended users and use environment. With the proper planning, scheduling, partner selection, and logistics, a successful international usability evaluation study can be achieved that optimizes ease of use, use safety, and effectiveness. 

---

Joe Pratt is a Certified Human Factors professional and the Lead Human Factors engineer at BlackHägen Design, an R&D and HFE consultancy. Experienced with devices ranging from respiratory care to interventional cardiology, Joe has contributed to the design of over 150 medical devices through usability research and usability consulting.