New computational techniques stemming out of the field of data science, including machine learning, computer vision, and artificial intelligence (AI) techniques (collectively, here—ML), stand poised to revolutionize clinical care. These include the decision making leading up to and surrounding clinical interventions, the decisions to use medical devices, and the ongoing therapies delivered via medical devices. While these techniques are powerful, becoming (or are, depending on industry) mainstream, and generally exciting, they are not without risk and can incur quite considerable time and capital costs to develop and deploy. In fact, as others have noted, the vast majority of measurables (cost, time, code, etc.) developed and incurred while developing any ML system are not the ML-algorithm itself.¹

There’s been a profound uptick in medical device companies making or wanting to make use of these ML techniques to meet unmet clinical needs and provide better therapies as these techniques may provide better and more adaptive insights than traditional methods. Meeting the need requires a focus on generalizing and standardizing the approach to developing ML-enabled technologies in the medical device space. By focusing on process, generalization, and re-use, it’s possible to reduce hidden time and capital costs in the development of image-based ML-algorithms.

Background
As an example, there have been many advancements in the neurointensive care pathway in recent decades, and both the neurosurgical and neurocritical care specialties are rapidly evolving. In our experience, new interventions that are developed are often surrounded by questions about their optimal usage: when? in whom? for how long? Therefore, coupling interventions with decision support leads to better outcomes for the patient (Figure 1). In this article, we examine a tool currently under development to support decisions surrounding the use of Minnetronix Medical’s newly FDA-cleared expandable port for deep brain access, called the MindsEye Expandable Port (Figure 2). This port, and others like it, are used to access a hematoma in a patient that has experienced an intracerebral hemorrhage (ICH), allowing for evacuation of the bleed. Accordingly, Minnetronix sought to develop a decision support tool that provides additional insight into the critical choice of surgical intervention (with minimally invasive tools like the MindsEye port) versus medical management in the ICH population.


The treatment paradigm of a patient with ICH is a subject of numerous recent and ongoing clinical trials.^2-4 The current standard of care ranges from aggressive evacuation of the blood to “watch and wait,” depending on size, location, severity, and other still-debated factors. Many of these factors may be clarified by careful, quantitative examination of the progression of the patient’s status on CT imagery. Unfortunately, this is time-consuming and not currently the standard of care through radiology. While there are manual tools to help evaluate some of the metrics of the CT image, there are, to date, no automated tools that cover the breadth of measures required.


With this in mind, Minnetronix sought to create an automated CT processing algorithm (CT segmentation) that calculates relevant anatomic and volumetric factors over time to assist the neurosurgeon in evaluating and deciding when and how to evacuate the hematoma (Figure 3). This algorithm, called the DepiCT Neuroimaging Platform, was developed using well-documented ML techniques. During the development of this algorithm, technical hurdles were encountered that are equally well-documented; as issues were resolved, the team focused on generalizable solutions and tool-creation to better serve medical device customers, who are increasingly asking for similar solutions to accompany their systems. Following are highlights of a few of these problems and the generalized solutions. Examples from each of six different phases of the ML algorithm development process are highlighted (Figure 4).


1. Data Acquisition
At the outset, we estimated that to develop a robust algorithm across numerous factors associated with the images or imaging equipment (e.g., technology, quality, time, anatomic, surgical), we would require thousands of CT scans from a variety of clinical sites. These scans would be labeled to create “ground truth” (gold standard) to train the ML algorithm. Timely, robust, and varied data collection is a well-known challenge in the biomedical space. According to both law and best practice, hospitals are protective of patient data. Data de-identification and accuracy must be top priority. To be successful and keep pace, the most effective tool was extreme flexibility. We collaborated with nearly a dozen sites. Some sites preferred to collaborate in the context of data-transfer agreements, while others preferred to operate as retrospective studies. We supported both approaches and followed the relevant processes to ensure success. In some cases, sites simply needed additional support navigating internal processes and procedures to expedite data transfer. Some sites preferred electronic transfer of imagery data via custom secured tools, some via standard cloud tools (e.g., Dropbox, Box, OneDrive, etc.), and others preferred USB sticks or even crates of DVDs, if it was easiest for them while following their internal processes to guarantee deidentification. We supported them all by setting up our own hosted many-terabyte datastore that could interact and receive any data electronically, via USB, as well as purchasing bulk DVD-rippers to create shorter work of the project.

2. Data Cleaning and Organization
One side effect of extreme flexibility during data acquisition is that this pushed the onus of cleaning (e.g., removing extraneous data) and organizing (creating a master database relating relevant clinical and technical factors to the imagery) to us. This was our preferred approach. We received data in DICOM format, as is typical for CT and most other medical imagery. This format is fraught with data management challenges and is best suited to reside within a PACS system database. Rather than being boxed in by off-the-shelf image acquisition systems, we developed our own internal tools to efficiently manage the data, tailored to the ML algorithm development workflow. As an example, while off-the-shelf DICOM viewer tools were useful for viewing the images and their associated metadata, they were not particularly useful for subsequent steps in the algorithm development lifecycle. Specifically, instead of utilizing a database approach, both our custom tool for manual segmentation for ground truth as well as the ML code for algorithm training operated better when organized as a hierarchy of human-readable folders on the filesystem, minimizing complexity and increasing ease-of-use of their implementations. As such, we used both customized off-the-shelf systems as well as wrote our own file management tools to bridge the gap between PACS database requirements and our filesystem needs, allowing us to accomplish the best of both worlds. Furthermore, as quickly as the process would allow, we moved away from the DICOM format to a more analysis-friendly format5,6 to facilitate data deployment. These tools are completely generic and able to handle other types of imagery for us or for our medical device company customers.

3. Ground Truth
The critical component of the type of ML algorithm that we developed for DepiCT was robust gold-standard “ground truth” data for training and against which the algorithm was tested. Without this, the algorithm would never train to appropriate performance levels. As previously mentioned, while there are manual tools available (with varying degrees of accuracy) as part of the standard clinical workflow for labeling a minority of the measures we trained for DepiCT, the majority of the measures are simply not performed today. Thus, there are no off-the-shelf tools that make the ground truth annotation process easy. Accordingly, Minnetronix developed its own tool to aid in the labeling of the ground truth for this algorithm. This is probably the most important time-saving step in the entire process since training of the algorithm required thousands of manually-labeled images. Without an efficient way to accomplish this, schedules and budgets could explode. To avoid this, we treated the development of our own, proprietary, manual segmentation tool as we would treat any clinical solution: by identifying the unmet need and the job that needs to be done. We focused on development of a general-purpose, readily-modifiable, and extensible manual image segmentation software suite; one that was purpose-built for the specifics of this challenge yet readily extensible to the next.

Furthermore, after experimenting with partners while labeling ground truth, we found that clusters of two-person teams greatly outperformed any one individual in labeling the features. We paired radiologists up with students trained as manual segmenters (Figure 5). The radiologists, trained to quickly read scan after scan in clinical practice, were very efficient at rapidly labeling features within a particular CT scan. Students were then brought in to refine what the radiologists had labeled to “pixel perfection.” The whole process was overseen by neuroradiology, to ensure the students interpreted the radiological inputs correctly. We therefore built a workflow management system into our tool, wherein scans would be routed to radiologists first for coarse labeling, and then to a student for polishing, with the capability for additional loops in the event of errors, tracking all interactions with the data and at a fraction of the cost of traditional approaches.


4. Algorithm Development
The development of the computational portion of the DepiCT algorithm is, of course, essential, and the aspect of this project that receives the most attention. ML-based techniques, as mentioned previously, are generally exciting, rapidly evolving, and becoming standard of medical care; however, they require specialists to harness their power. We recognized quickly that the level of specificity for this program required such specialists, and we supplemented our team with the required expertise. While this addition allowed us to move quickly from concept to proof of principle, optimizing the logistics was key to that success. We found if we provided clean, organized, robustly-segmented ground-truth data to our algorithmic team, they could quickly and cost-effectively guide us through the process of training and testing the DepiCT algorithm, greatly increasing the likelihood of an effective and refined product. We found, upon review, a very similar trend to what others had published¹—in terms of scope of effort (code, time), the actual training of the algorithm is dwarfed by all other steps in this process. Hence, the additional care in leaning out and generalizing the tools greatly impacted our capability to control cost and schedule. Further, the tools created to train the algorithm are themselves reasonably general-purpose for solving related medical imaging problems.

5. Data Deployment
The team of radiologists and students that worked to establish ground truth were a truly global lineup. While this global collaboration provided reliable data in a controlled and secured fashion, it also created challenges. We did not want to depend upon, or make assumptions about, their hardware or their internet bandwidth, leading us to a cloud-based solution for manual segmentation. However, we also wanted to avoid the expense or security risks of transitioning our custom Microsoft Windows-based manual segmentation tool to a third-party cloud environment. Thus, we compromised and optimized by standing up our own “pseudo-cloud.” We used our own VPN and remote desktop environment for the manual segmenters, allowing us to centralize the thousands of images on our local datastore. Conversely, our algorithm development partners required data locally on their filesystem to most expediently train the algorithm, and we required a mirror of that data for testing purposes locally. For this, we relied on more traditional data transfer and mirroring tools.

6. User Feedback
Lastly, it cannot be overstated how important it is, for any medical device under development, to solicit feedback from the intended users of the product during the development process. In our scenario, we wanted to get quick hands-on user feedback of the DepiCT system’s capabilities and utility. However, we did not want to incur the full expense of building out the entire software architecture to show the final product in a cloud setting. Typically, this problem could be addressed by going directly to the customer to meet, observe, and obtain feedback. This can be a slow process for a small company and, in our case, was made near-impossible due to COVID-19 restrictions. We required an interface that would be easy to interact with, require no installation on the user’s computer, and be available both on- and offline to allow the user to interact with the system on their own time and terms.

Our solution surprised us. After some investigation, we learned that it is a fairly straight-forward exercise to automate the creation of PowerPoint slides, complete with intra-presentation navigation buttons to simulate a user interface, using various scripting tools. This met all of our requirements: no installation, near-universal availability, on- and offline, and an intuitive process. Further, we minimized the complexity of our automated scripts by carefully choosing which interactions to expose to the user; that is, which buttons gave us the most feedback and were most impactful to the user’s experience. We were thus able to work with customers in real-time with standard screen sharing tools or allow them to interact offline and provide feedback at their own pace.

Conclusion
Machine learning techniques are enabling a new type of decision support paradigm that stands to uplift and enable utilization of traditional medical devices. We believe this to be true as we develop and plan to commercialize our own tool to assist clinicians caring for patients who have suffered an ICH. We also are finding this to be true more broadly with customers for whom we develop medical technologies. While developing our own solution, we have created a set of multipurpose tools that offer optimization, efficiency, and reusability, and we’ve been pleasantly surprised at how cost-effective our approach has become. We believe this will ultimately reduce time to market of these important new capabilities for us as well as for our medical device company customers. 

References

1. bit.ly/mpo210151
2. bit.ly/mpo210152
3. bit.ly/mpo210153
4. bit.ly/mpo210154
5. bit.ly/mpo210155
6. bit.ly/mpo210156

Caution: Federal (U.S.A.) Law restricts the sale of the MindsEye device to sale by or on the order of a physician.
Caution-Investigational Device, DepiCT Informatics Platform is limited by Federal (or U.S.) Law to Investigational Use and is not available for sale in the U.S. or outside the U.S.

---

Aaron McCabe, Ph.D., is the director of research and technology at Minnetronix Medical. He has over two decades experience in a variety of roles in the medical device industry. Since 1996, Minnetronix Medical has accelerated medical device breakthroughs as a design, development, and manufacturing partner to leading device companies around the world. From design to commercialization to whole product solutions, Minnetronix brings expansive industry insight and intentional technical acumen, delivering better medical devices to market, faster. For more information, visit bit.ly/3sUJTq2.