07.26.10
Technical Debt in Medical Software
Early intervention leads to longer product life.
Dan O’Connor, Tim Bowe
Foliage Software Systems Inc.
The software development community is known for inventing useful metaphors as a means to relate technical topics to “real world” concepts. The “technical debt” metaphor was introduced first by Ward Cunningham (an American computer programmer who developed the first wiki website) to compare deficiencies in software with financial debt, a topic that is now all too familiar for most of us.1 Just as the United States will pay the penalty for excessive financial debt with slower growth and less economic competitiveness for the next few years, excessive technical debt has the impact of slower development velocity and reduced product competitiveness.
There are many situations where a development team will decide to ship a product with known deficiencies in the software design and/or implementation. This can be the proper thing to do under certain circumstances. The application may be feature complete, and functionally correct, yet there are known shortcuts that have been taken during development. Better design ideas always are much easier to see in hindsight, and good developers usually are eager to get it perfect. Inevitably the question arises: How done is done?
This scenario happens all the time—the code freeze date was announced months ago, and there is a scramble to get the feature enhancements completed. Consequently, additional shortcuts are taken. Finally, the decision is made to move forward. Ship it! Any additional improvements will have to wait until later. But “later” too often translates to “never”; it is all too easy to forget about those shortcuts and deficiencies after the product is released and the organization moves on to product support and enhancement activities. This is an example of technical debt; it must be recognized, and it will be serviced one way or another. Other examples of technical debt are products with insufficient architectures to support the overall product goals over time or the design degradation of programs through poor feature implementation or program maintenance.
The technical debt metaphor turns out to be a useful vehicle for communication between product development teams and executive management, who otherwise may not be eager to invest resources in cleanup tasks, redesign or refactoring. Servicing debt, however, is a concept that management does understand, and its importance tends to sink in when we can identify the specific costs associated with long-term, unattended debt. As with financial loans, technical debt has the phenomenon of compounding interest; left unattended it becomes increasingly expensive. There must be a plan to track the technical debt, to deal with the interest, and eventually to pay down the principal.
Deliberate and Inadvertent Debt
Martin Fowler, an author and international speaker on software development, adds an interesting perspective with his classification of technical debt into quadrants.2 His suggestion is to distinguish between deliberate and inadvertent technical debt as well as between prudent and reckless debt.
It’s likely that many medical software systems have accumulated technical debt over years of successful product development. As is the case with financial loans, incurring technical debt can be an appropriate strategy when used prudently. In our experience, the amount of technical debt evidenced in medical software roughly has correlated to the medical device classifications. Class III devices, such as pacemakers and implantable defibrillators, thankfully, tend to have relatively little accumulated technical debt associated with them. The design controls required to ensure safety and effectiveness at this level generally prevent accumulation of serious technical debt.
Moving toward the lower device classifications, we have seen more evidence of technical debt, and under certain conditions this can be a prudent decision. As an example, suppose a patient monitoring vendor needs to hit a key market window. They may decide to postpone some beneficial software refactoring in order to hit the deadline. It also is quite possible for organizations to accumulate inadvertent technical debt over time. This likely would be the result if your software development team has experienced staff reductions or turnover across product releases.
Strategic Response to the New Landscape
It’s wise to spend some time assessing your technical debt before any major product development program begins. In the area of medical software, there are several factors that currently are motivating such a surge in strategic development initiatives.
One important driver that is influencing nearly all of the health information technology companies is the set of directives that came out of the American Recovery and Reinvestment Act—otherwise known as the federal government’s recent economic stimulus package—last year. Planned feature initiatives have been disrupted in many cases in a scramble to deliver certified products containing required feature sets that will allow their customers to realize the promised financial incentives.
A second influential trend is additional regulatory pressure. Until recently, most healthcare information technology (HIT) systems have not been assigned a medical device classification. This is now changing as the U.S. Food and Drug Administration has begun to inform companies that their HIT systems (or key aspects of them) fall within the definition of a medical device. In order to satisfy the regulatory burden, vendors will need to address not only technical debt within the code base, but also process, documentation and traceability debt.
A third driver impacting medical product development is increasing financial pressure on hospitals and health care providers. The need to continually “do more with less” has accelerated consolidation among care providers and has influenced the creation of large care delivery networks. Fewer buyers create additional competitive pressure and will eventually mean fewer vendors.
In severe cases, the technical debt problem is compounded when an organization has grown through technology acquisition. In our experience, it is the rare company that has done a stellar job in technology and organizational assimilation. More commonly, we see that organizational “silos” remain years after the acquisitions. It is not uncommon for an organization to be maintaining and extending two, three or more separate technology platforms. Depending on the vintage of the technology and the accumulated technical debt, it might be the case that none of the legacy platforms is well positioned to satisfy the new drivers identified in the forward-looking product roadmap.
Assessing Technical Debt
Technical debt can manifest itself in many different ways. But there are several familiar clues to look for, and tools also can help identify potential problem areas.
Complexity: Complex code is a fundamental sign of technical debt. Many legacy software systems have a few infamous code modules that are difficult to maintain. If this situation resonates, then you have experience with technical debt.
Code Duplication: When a code freeze is approaching, “cut and paste” can be a favorite approach in the developer’s bag of tricks. This is classic technical debt—shortcut now, pay for your sins later. Duplicate code is a well-known path to exploding code size, increasing overall complexity and introduction of bugs in future releases. Static analysis tools do a good job of locating the duplicate code blocks for you.
Documentation Debt: Developers are not known for being enamored with documentation in the first place. Lack of documentation does not directly affect the runtime characteristics of the software system, but missing design or test documentation needs to be counted as technical debt because there is interest to be paid. This cost will show up on subsequent releases when a new developer is charged with updating the undocumented modules.
Testing Debt: Unit testing is another item that gets squeezed due to schedule pressure. The interest on this type of debt is the lack of a safety net to catch regressions. Current best practices in software development call for unit tests for all modules and for running all tests during continuous integration. Noted software developer, consultant and author Michael Feathers calls this “code that bites back,” meaning that the system informs you (usually by email) if your submission just broke a unit test. Again, good code coverage tools exist to help you understand your testing debt.3
Architectural Debt: Architectural debt exists when the software architecture is no longer well aligned with the key product drivers, and this can happen in two ways. Developers can cause architectural decay if they do not understand and follow the architecture rules and maintain the intended conceptual integrity. The second path to architectural debt is when the product itself evolves well past the capability covered by the original architectural design.
Tracking Technical Debt
Perhaps the simplest method of tracking technical debt is to add commentary in the code itself. Adding “to-do” comments is a useful practice that has been around forever. We know one senior developer who puts a personal touch on his tracking comments that read “worry about x” and “worry about y.”
Some agile development teams use informal debt statements, written on index cards and posted on a bulletin board. Other organizations use a more formal tracking method, which calls for developers to enter debt issues into the tracking system. This allows the organization to prioritize the issue and estimate the time for the required refactoring. It also allows the debt issues to be added to the schedule and to be managed like any other development task.
Another organization we worked with established a simple but effective “pain dashboard” to track their technical debt. This was manifest as a wiki page where developers easily could add entries describing the parts of the system that were difficult to understand and were debilitating to their productivity. The system allowed for a voting scheme so the team had a self-balancing mechanism to continually prioritize their technical debt.
Bankruptcy as the Last Resort
When faced with crippling financial debt, bankruptcy always is the last resort. We can extend the technical debt metaphor nicely here. For software intensive systems, bankruptcy occurs when you conclude that the legacy code base is a dead end and is no longer viable to support the forward looking product roadmap. A complete rewrite and a new platform are needed for the organization to be competitive going forward. In this case the technical debt can be retired along with the legacy system, and like filing Chapter 11, you are no longer responsible to address all the sins of the past.
Initially, this may sound like an attractive option, but the decision to undertake a rewrite never should be made lightly. Developers rarely get the opportunity for a clean sheet redesign, and for good reason; it is expensive. Developing a new software platform can represent an investment of perhaps millions of dollars. On the other hand, a redesign represents an opportunity to leverage newer, better tools to significantly increase the productivity of your software staff (with a resultant cost reduction) as well as a chance to retire a decade of technical debt en masse.
Benefits of Paying Down the Debt
Software-intensive medical products have, more likely than not, incurred technical debt over the course of a successful product life cycle. Take time to pay down the technical debt and your organization will benefit by:
Increased R&D efficiency and improved time to market: Once the code base has been cleared of crippling technical debt, the development velocity will increase. The design refactoring is effectively cleaning the molasses out of your software development machinery.
Hitting commitment dates: Paying down the technical debt will increase the team’s overall understanding of the code base. This will have the effect of getting better estimates for work to be completed and reducing risk during subsequent modifications.
Performance and technology upgrades: Another positive side effect of attacking the technical debt is the opportunity to upgrade to the latest technologies. This can mean moving to the latest versions of compilers and development tools. It also can mean upgrading to the latest versions of the operating system and third-party libraries and possibly even to a faster processor.
It is important to be aware of the impact of technical debt as product offerings are extended to support a strategic product roadmap. Code evaluations and architecture assessments can help to identify the technical debt. The assessment of technical debt must be put in context with the organization’s forward-looking product strategy, as emerging market trends are addressed and the challenges of an increasingly competitive landscape are met.
References:
1. Ward Cunningham,
http://c2.com/cgi/wiki?TechnicalDebt
2. Martin Fowler,
http://martinfowler.com/bliki/TechnicalDebtQuadrant.html
3. Michael Feathers. Working Effectively with Legacy Code. Prentice Hall, Englewood Cliffs, N.J., 2005.
As CEO of Foliage, a software development and consulting firm in Burlington, Mass, Tim Bowe steers the development and evolution of the company’s consulting services, and works directly with clients, helping them derive more business value from their product lines. Tim also speaks and writes extensively on leveraging product line synergies to create market opportunity, and on increasing R&D efficiency through focused alignment of technology and business strategies. Prior to Foliage, Tim held senior management and technical roles at Technology Integration, Inc., Eaton Semiconductor, Advanced NMR, Ion Beam Systems and Control Data. He holds a bachelor’s degree in electrical engineering from Merrimack College, and did graduate work in physics and computer science at the University of New Hampshire.
Dan O’Connor is a software architect for Foliage with more than 25 years of software development experience. He has been with Foliage for 14 years, and his current position includes roles as project manager, software architect and lead consultant. O’Connor has worked on many mission/safety critical software programs at Foliage including applications in medical systems, semiconductor manufacturing systems, and aviation systems. Prior to joining Foliage, O’Connor was a principle software engineer at Computervision Corp., where he spent 11 years developing CAD/CAM products. O’Connor holds a MSCS from Boston University and a BSCS from University of Massachusetts at Lowell.
Early intervention leads to longer product life.
Dan O’Connor, Tim Bowe
Foliage Software Systems Inc.
The software development community is known for inventing useful metaphors as a means to relate technical topics to “real world” concepts. The “technical debt” metaphor was introduced first by Ward Cunningham (an American computer programmer who developed the first wiki website) to compare deficiencies in software with financial debt, a topic that is now all too familiar for most of us.1 Just as the United States will pay the penalty for excessive financial debt with slower growth and less economic competitiveness for the next few years, excessive technical debt has the impact of slower development velocity and reduced product competitiveness.
There are many situations where a development team will decide to ship a product with known deficiencies in the software design and/or implementation. This can be the proper thing to do under certain circumstances. The application may be feature complete, and functionally correct, yet there are known shortcuts that have been taken during development. Better design ideas always are much easier to see in hindsight, and good developers usually are eager to get it perfect. Inevitably the question arises: How done is done?
This scenario happens all the time—the code freeze date was announced months ago, and there is a scramble to get the feature enhancements completed. Consequently, additional shortcuts are taken. Finally, the decision is made to move forward. Ship it! Any additional improvements will have to wait until later. But “later” too often translates to “never”; it is all too easy to forget about those shortcuts and deficiencies after the product is released and the organization moves on to product support and enhancement activities. This is an example of technical debt; it must be recognized, and it will be serviced one way or another. Other examples of technical debt are products with insufficient architectures to support the overall product goals over time or the design degradation of programs through poor feature implementation or program maintenance.
The technical debt metaphor turns out to be a useful vehicle for communication between product development teams and executive management, who otherwise may not be eager to invest resources in cleanup tasks, redesign or refactoring. Servicing debt, however, is a concept that management does understand, and its importance tends to sink in when we can identify the specific costs associated with long-term, unattended debt. As with financial loans, technical debt has the phenomenon of compounding interest; left unattended it becomes increasingly expensive. There must be a plan to track the technical debt, to deal with the interest, and eventually to pay down the principal.
Deliberate and Inadvertent Debt
Martin Fowler, an author and international speaker on software development, adds an interesting perspective with his classification of technical debt into quadrants.2 His suggestion is to distinguish between deliberate and inadvertent technical debt as well as between prudent and reckless debt.
It’s likely that many medical software systems have accumulated technical debt over years of successful product development. As is the case with financial loans, incurring technical debt can be an appropriate strategy when used prudently. In our experience, the amount of technical debt evidenced in medical software roughly has correlated to the medical device classifications. Class III devices, such as pacemakers and implantable defibrillators, thankfully, tend to have relatively little accumulated technical debt associated with them. The design controls required to ensure safety and effectiveness at this level generally prevent accumulation of serious technical debt.
Moving toward the lower device classifications, we have seen more evidence of technical debt, and under certain conditions this can be a prudent decision. As an example, suppose a patient monitoring vendor needs to hit a key market window. They may decide to postpone some beneficial software refactoring in order to hit the deadline. It also is quite possible for organizations to accumulate inadvertent technical debt over time. This likely would be the result if your software development team has experienced staff reductions or turnover across product releases.
Strategic Response to the New Landscape
It’s wise to spend some time assessing your technical debt before any major product development program begins. In the area of medical software, there are several factors that currently are motivating such a surge in strategic development initiatives.
One important driver that is influencing nearly all of the health information technology companies is the set of directives that came out of the American Recovery and Reinvestment Act—otherwise known as the federal government’s recent economic stimulus package—last year. Planned feature initiatives have been disrupted in many cases in a scramble to deliver certified products containing required feature sets that will allow their customers to realize the promised financial incentives.
A second influential trend is additional regulatory pressure. Until recently, most healthcare information technology (HIT) systems have not been assigned a medical device classification. This is now changing as the U.S. Food and Drug Administration has begun to inform companies that their HIT systems (or key aspects of them) fall within the definition of a medical device. In order to satisfy the regulatory burden, vendors will need to address not only technical debt within the code base, but also process, documentation and traceability debt.
A third driver impacting medical product development is increasing financial pressure on hospitals and health care providers. The need to continually “do more with less” has accelerated consolidation among care providers and has influenced the creation of large care delivery networks. Fewer buyers create additional competitive pressure and will eventually mean fewer vendors.
In severe cases, the technical debt problem is compounded when an organization has grown through technology acquisition. In our experience, it is the rare company that has done a stellar job in technology and organizational assimilation. More commonly, we see that organizational “silos” remain years after the acquisitions. It is not uncommon for an organization to be maintaining and extending two, three or more separate technology platforms. Depending on the vintage of the technology and the accumulated technical debt, it might be the case that none of the legacy platforms is well positioned to satisfy the new drivers identified in the forward-looking product roadmap.
Assessing Technical Debt
Technical debt can manifest itself in many different ways. But there are several familiar clues to look for, and tools also can help identify potential problem areas.
Complexity: Complex code is a fundamental sign of technical debt. Many legacy software systems have a few infamous code modules that are difficult to maintain. If this situation resonates, then you have experience with technical debt.
Code Duplication: When a code freeze is approaching, “cut and paste” can be a favorite approach in the developer’s bag of tricks. This is classic technical debt—shortcut now, pay for your sins later. Duplicate code is a well-known path to exploding code size, increasing overall complexity and introduction of bugs in future releases. Static analysis tools do a good job of locating the duplicate code blocks for you.
Documentation Debt: Developers are not known for being enamored with documentation in the first place. Lack of documentation does not directly affect the runtime characteristics of the software system, but missing design or test documentation needs to be counted as technical debt because there is interest to be paid. This cost will show up on subsequent releases when a new developer is charged with updating the undocumented modules.
Testing Debt: Unit testing is another item that gets squeezed due to schedule pressure. The interest on this type of debt is the lack of a safety net to catch regressions. Current best practices in software development call for unit tests for all modules and for running all tests during continuous integration. Noted software developer, consultant and author Michael Feathers calls this “code that bites back,” meaning that the system informs you (usually by email) if your submission just broke a unit test. Again, good code coverage tools exist to help you understand your testing debt.3
Architectural Debt: Architectural debt exists when the software architecture is no longer well aligned with the key product drivers, and this can happen in two ways. Developers can cause architectural decay if they do not understand and follow the architecture rules and maintain the intended conceptual integrity. The second path to architectural debt is when the product itself evolves well past the capability covered by the original architectural design.
Tracking Technical Debt
Perhaps the simplest method of tracking technical debt is to add commentary in the code itself. Adding “to-do” comments is a useful practice that has been around forever. We know one senior developer who puts a personal touch on his tracking comments that read “worry about x” and “worry about y.”
Some agile development teams use informal debt statements, written on index cards and posted on a bulletin board. Other organizations use a more formal tracking method, which calls for developers to enter debt issues into the tracking system. This allows the organization to prioritize the issue and estimate the time for the required refactoring. It also allows the debt issues to be added to the schedule and to be managed like any other development task.
Another organization we worked with established a simple but effective “pain dashboard” to track their technical debt. This was manifest as a wiki page where developers easily could add entries describing the parts of the system that were difficult to understand and were debilitating to their productivity. The system allowed for a voting scheme so the team had a self-balancing mechanism to continually prioritize their technical debt.
Bankruptcy as the Last Resort
When faced with crippling financial debt, bankruptcy always is the last resort. We can extend the technical debt metaphor nicely here. For software intensive systems, bankruptcy occurs when you conclude that the legacy code base is a dead end and is no longer viable to support the forward looking product roadmap. A complete rewrite and a new platform are needed for the organization to be competitive going forward. In this case the technical debt can be retired along with the legacy system, and like filing Chapter 11, you are no longer responsible to address all the sins of the past.
Initially, this may sound like an attractive option, but the decision to undertake a rewrite never should be made lightly. Developers rarely get the opportunity for a clean sheet redesign, and for good reason; it is expensive. Developing a new software platform can represent an investment of perhaps millions of dollars. On the other hand, a redesign represents an opportunity to leverage newer, better tools to significantly increase the productivity of your software staff (with a resultant cost reduction) as well as a chance to retire a decade of technical debt en masse.
Benefits of Paying Down the Debt
Software-intensive medical products have, more likely than not, incurred technical debt over the course of a successful product life cycle. Take time to pay down the technical debt and your organization will benefit by:
Increased R&D efficiency and improved time to market: Once the code base has been cleared of crippling technical debt, the development velocity will increase. The design refactoring is effectively cleaning the molasses out of your software development machinery.
Hitting commitment dates: Paying down the technical debt will increase the team’s overall understanding of the code base. This will have the effect of getting better estimates for work to be completed and reducing risk during subsequent modifications.
Performance and technology upgrades: Another positive side effect of attacking the technical debt is the opportunity to upgrade to the latest technologies. This can mean moving to the latest versions of compilers and development tools. It also can mean upgrading to the latest versions of the operating system and third-party libraries and possibly even to a faster processor.
It is important to be aware of the impact of technical debt as product offerings are extended to support a strategic product roadmap. Code evaluations and architecture assessments can help to identify the technical debt. The assessment of technical debt must be put in context with the organization’s forward-looking product strategy, as emerging market trends are addressed and the challenges of an increasingly competitive landscape are met.
References:
1. Ward Cunningham,
http://c2.com/cgi/wiki?TechnicalDebt
2. Martin Fowler,
http://martinfowler.com/bliki/TechnicalDebtQuadrant.html
3. Michael Feathers. Working Effectively with Legacy Code. Prentice Hall, Englewood Cliffs, N.J., 2005.
As CEO of Foliage, a software development and consulting firm in Burlington, Mass, Tim Bowe steers the development and evolution of the company’s consulting services, and works directly with clients, helping them derive more business value from their product lines. Tim also speaks and writes extensively on leveraging product line synergies to create market opportunity, and on increasing R&D efficiency through focused alignment of technology and business strategies. Prior to Foliage, Tim held senior management and technical roles at Technology Integration, Inc., Eaton Semiconductor, Advanced NMR, Ion Beam Systems and Control Data. He holds a bachelor’s degree in electrical engineering from Merrimack College, and did graduate work in physics and computer science at the University of New Hampshire.
Dan O’Connor is a software architect for Foliage with more than 25 years of software development experience. He has been with Foliage for 14 years, and his current position includes roles as project manager, software architect and lead consultant. O’Connor has worked on many mission/safety critical software programs at Foliage including applications in medical systems, semiconductor manufacturing systems, and aviation systems. Prior to joining Foliage, O’Connor was a principle software engineer at Computervision Corp., where he spent 11 years developing CAD/CAM products. O’Connor holds a MSCS from Boston University and a BSCS from University of Massachusetts at Lowell.
