The Butterfly Effect of AI on Cybersecurity

I have very intentionally avoided writing about artificial intelligence (AI) and large language models (LLMs), because it is being addressed by everyone, usually with something to sell. In addition, AI is changing so fast that publishing delays of a couple of months means pretty much anything that gets written will already be old by the date of publication. But I do think I need to address it, at least in a very broad sense.

It is already quite evident that, unlike other categories of computer-based tooling that have emerged over the course of the previous five decades, AI is changing everything about medical device development and cybersecurity. Further, its impact isn’t being felt by just one aspect of development (such as coding), it is affecting the entire product development lifecycle.

AI is going to make a huge impact on our world as we know it. As such, I find it humorous when I read about someone saying it won’t do this or that, only for the next week to see the introduction of a new AI tool that accomplishes the task the naysayers just said was impossible. The impact will be so far-reaching it will be difficult to ascertain the full breadth of its significance.

The Butterfly Effect

First, let’s examine an area far outside of medical device development. I recently read an article about the use of AI-based cameras to replace smoke spotters (rangers who sit atop mountains to spot early forest fires).¹ Since AI is cheaper, easier to manage, and better at detecting fires earlier, the era of smoke spotters stationed on mountain peaks is rapidly coming to an end.

As a result, those rangers will no longer have a reason to be employed and a high percentage of them will be laid off. This is a first-order impact, and, rather obviously, going forward, there will be no need for as many rangers. The rangers can be seen as analogous to software engineers.

Before AI, the smoke spotters needed a tower and a small building in which to live on top of their respective mountain. Helicopter trips (roads do not extend to these mountain tops) to deliver supplies and construction teams to build the aforementioned structures will no longer be required. These losses are referred to as second-order impacts on the community. They can be viewed as analogous to many roles, including middle managers, commercial building leases, software development tools, cybersecurity tools, software and product testing, and electronic hardware development.

Third order effects encompass factors once more removed. These would include more crime (desperate people with time on their hands), fewer to no field repair staff (for when drones flying up to replace the intelligent camera systems fail), shuttering of helicopter companies (those that existed significantly on the revenue from the aforementioned deliveries), and fewer fires (leading to an increase in forest debris, and thus, a higher rate of illness among the trees). These would be analogous to a lack of systems engineering, as spec designed AI guiderails are created (even then, the specs are AI generated); a general malaise among people with illnesses, due to a lack of humanity in their treatment; a general lack of caring about the outcome for a patient; and no field service, as the medical devices will be cheap, easily replaceable, disposable edge devices that communicate with a backend AI system. This scenario creates a further divide between the “haves” and the “have-nots” for medical care.

This example presented a significant level of effect that was due to a simple AI camera being used in place of a human. Small changes can make huge ripples—the butterfly effect. If you can affect forest rangers, helicopter pilots, and construction workers all in the same breath with the same tool, it’s making a broad brushstroke that’s impossible to predict.

What will these technologies do to our society? The effects are so wide-ranging and deeply impactful, it is difficult to see a positive outcome.

Already, you’re seeing fallout in unlikely places. For example, China’s court recently ruled firms can’t lay off workers due to AI replacements.² That’s in China, where concerns for the human worker and the individual does not always seem to be a priority. What will the fallout be in the Western world?

• Will there be similar legal actions?
• Will the working class be relegated to a lower status, since they will lack value?
• Will widespread depression grow among the population?
• Will people be paid by the government to support basic living needs (and where does that end)?
• Could we see mass die-offs of the population due to famine, disease, and the inability to support the current population level when the masses are not needed?

All these scenarios are possible; it simply depends on how we manage them moving forward. Of course, management of these technologies is not being considered, as the rush to make money is all-consuming.

The Impact on Cybersecurity 

What does all this mean for cybersecurity, in particular, medical device cybersecurity? AI and LLMs will bring seismic changes to how we develop and support medical devices, including the cost, capabilities, and timelines previously required.

All of this revolves around time. Everything related to AI is about time; the adoption of AI has happened too quickly, the changes to how we follow best practices have changed too quickly (when it comes to developing with AI, we don’t even know what best practices are as they seem to change each week), and the products we are creating have been developed too quickly without the normal oversight time allows. Corporations are not known for rapidly responding to a changing environment. This is going to be a significant challenge and may ultimately destroy many of the larger established companies that cannot quickly adopt AI into all aspects of medical device development.

These tools haven’t even yet been used to develop the next generation of AI/LLM applications; the virtual toes have only just been dipped into the water. However, within the next year or two, we’re going to see AI tools that have been developed primarily by other AI tools. This already incredible pace of change will increase exponentially with these new releases. 

Giving names to new versions like Sonnet, Opus, or Mythos will be completely impractical. These models are going to change at such an accelerated rate that there is no way you will be doing any versioning on them. I doubt we will even see any sort of semantic versioning applied, let alone specific names assigned. 

Currently, a substantial amount of software and firmware can be written by LLMs. They do a very good job of writing code, provided you are a skilled AI user and know how to craft the prompts and create guardrails for software development.

Who does this impact? Does it impact the intern, the junior software engineer? No. Right now, those people have been kept in place while the expensive senior people are being let go. This is exactly the opposite of what will eventually start to happen. This is where the mistake of letting senior engineers go will become apparent; these are the folks who actually understand how the system was designed to function. A sudden reversal of retention policies will prioritize senior staff who can provide a “human in the loop” review of LLM-created products.

While today, these LLMs are reasonably sufficient at coding, that’s approximately where their impact ends. Currently, other areas related to cybersecurity (such as architecture; some firmware, such as timing-critical code; and the supply chain) remain largely untouched. This includes downstream activities from the manufacturer, such as production processes and ensuring a product is delivered malware-free. At the moment, AI doesn’t wade into those segments. 

There are early use cases of AI being used for threat modeling in design, but again, it is not verifying a secure design. It simply speeds up the process of identifying problems. 

We already know LLMs do a great job of finding zero-day vulnerabilities in our hardware and software. This can pose a challenge going forward as we try to use standardized commercial platforms (such as operating systems and electronics) in our devices. Look no further than the use of Windows in medical devices to see how it is the single weakest link in any such system.

So, what does this mean for LLMs? LLMs are adept at attacking both novel, bespoke solutions (custom-built platforms and operating systems) and commercial, off-the-shelf offerings. They will provide a leveling of the playing field in terms of system design choices.

Certainly, using legacy products developed 15 years ago and trying to incorporate them into the latest version of a device is going to be impossible. This is where things start to clash with the bean counters and their business model. Keeping elements around for 10, 20, or 30 years and reusing old designs and code is no longer viable.

Is this new dynamic going to be recognized by all businesses? Probably not. They are going to be very myopic in their view of this change. They’re not going to want to address it, and they certainly won’t want to spend their money to secure things.

What we’re going to see is quite a shakeout in the industry as these companies become pariahs. We have done a poor job of addressing the low-hanging fruit of vulnerabilities over the last 20 years. Now, those low-hanging fruits won’t be the only concern. All fruits, both high-level and low-level, are going to be exploited, and thus, addressed equally and rapidly, much faster than we’ve ever seen before.

Looking Ahead

There are going to be winners and losers in this race involving the use of AI and LLMs. Today, many who are using it have a chatbot build some code and cut and paste it into their project. They are just getting their toes wet. Unless you are altering the entire company structure (e.g., how it works, how it performs, and, perhaps most importantly, the speed at which it performs) to align with AI and what it will bring tomorrow, you are going to be one of the companies left behind.

I think we are going to start seeing two major changes. 

• Large “glacier” organizations that cannot adapt to new paradigms will start to lose tremendous amounts of market share and business (it’s already taking place for some).
• Smaller, leaner companies will emerge that can change and become AI-controlled organizations.

These more agile organizations are the ones that will become the new segment leaders within the next five years. 

Eventually, highly experienced engineers with multidisciplinary backgrounds in hardware, software, firmware, and systems engineering will leverage these tools to accelerate the success of the organization. They will guide the LLMs on what is being produced and, most importantly, review the output. “Human in the loop” will have to be with us for many years to come, as we cannot yet trust these systems. In fact, I don’t think we’re anywhere close to a world in which we can trust the outputs of these LLMs without a human in the loop (certainly not for medical devices).

This means all the junior level software engineers trying to learn and rise through the ranks to be the next generation of highly skilled, multidisciplinary engineers aren’t going to be there. We’re going to have a collection of old, expensive folks sitting around doing this limited work. Further, since we will not be creating the next generation of software/hardware/systems experts, when this generation “ages out,” the AI will be taking over all of it.

Hopefully, by then, the AI-developed AI systems will be sufficiently capable, and cybersecurity will be considered at each step of product creation, including design, implementation, and testing. It is certainly my hope this will be the case, because regardless, that is where we are headed. 

Final Thought

Will AI create some amazing new medical achievements? Absolutely! However, the journey to get there is terrifying, especially when you consider AI’s use will be governed by money, not by compassion for your fellow humans.

References
¹ tinyurl.com/mpo260631
² tinyurl.com/mpo260632

MORE FROM THIS AUTHOR—To Provide or Not: Bring Your Own Device vs. Kitted

Christopher Gates is the founder and CEO of arsMedSecurity, a medtech cybersecurity consulting firm. He is a recognized thought leader in medical device cybersecurity and the current co-chair for H-ISAC’s MDSC. Gates has over 50 years of experience developing and securing medical devices and works with industry-leading device manufacturers. He frequently collaborates with regulatory and standard bodies, including the CSIA, Health Sector Coordinating Council, H-ISAC, and Bluetooth SIG.