How Do Your Costs Compare to Your Competition?
This column installment wraps up a four-part series of articles tackling different aspects of software development. Part one (in the March issue of Medical Product Outsourcing) introduced the concept of “software inventory,” which was defined as the number of lines of source code that are an essential ingredient in a company’s shipped products. As is the case with many companies’ strategic assets, this software inventory has both an acquisition cost and an ongoing maintenance cost. Part two of the series (in the June issue of MPO) described approaches to calculating these costs, and understanding the underlying factors behind them. Part three (in theSeptember issue) explored some of the factors that affect costs in different companies and industries. This forth and final column will provide examples of how the techniques described can be applied, and the consequences of either overinvesting or underinvesting in your software inventory.
An Inventory Size andEffort Example
Some of the most difficult software to write and maintain is the software embedded in large capital equipment products. Examples can be found in every domain, from magnetic resonance imaging machines in hospitals to process tools in semiconductor fabs. Whatever the domain, software in machines tends to have a complex architecture that includes control algorithms, error handling and diagnostics, user interfaces, and databases.
Much of our software development work in the past 20 years has been in control systems for this type of large capital equipment, and many of the concepts described in this article have come from that work.
The following example shows how a customer applied these concepts.
The example company sells a wide product line of major medical equipment and associated medical information systems. The products total about 3 million lines of source code as counted using rules presented in an earlier installment of this series. The company has about 95 staff members in software research and development, including test staff and management, once again using rules described in an earlier article.
At $150,000 fully burdened per R&D staff member per year, and at 400 lines per staff-month as estimated by the company in collaboration with Foliage, this comes out to a total retrospective investment to date by the company of 625 staff-years and $93.8 million. Thus, each line of code has cost the company an average of $31 to develop and maintain during that period.
Equipment control systems are harder to develop than average software, and medical systems require significantly more testing than most other kinds of software. Therefore, one would expect that this staff’s average productivity should be significantly lower than the software industry average of about 700 lines per month. This staff’s productivity of 400 lines per staff-month is actually somewhat higher than what we typically find within medical, in part because not all of the software has yet come under regulatory guidelines. These numbers suggest that the company consistently isinvesting appropriately in software architecture, design, implementation and, in particular, testing—which indeed turned out to be the case.
The company’s software R&D staff averaged around 95 permanent employees across the typical 10-year life cycle of any of the individual products. Each employee was responsible on average for approximately 32,000 lines of code, which is lower than industry numbers, but well within the norm for medical or aerospace. This amount of code per staff member suggests that the company is investing appropriately in the long-term maintenance of its software inventory. A significantly higher number of lines of code per developer would suggest that code improvements and bug fixing are being deferred.
In this example, each individual software product was shipped to its first paying customer after two to three years of development effort, and the original development team for each individual product was five to 15 staff. For about two years after the first shipment of each individual product, almost the whole team continued to work full time on the software as it matured in features and stability, and as more and more customers began using it. Beyond the two-year mark, staffing on each product historically has tailed off over time, more or less consistently with Figure 1 (above), which was introduced in the second article in this series.
Under-Producing, Over-Producing, and Technical Debt
The software development team in this example is typical in both the number of lines per month and in the overall number of lines of code supported by each developer, given the specific demands of medical software. But what does it mean when a company measures its inventory costs and finds that they are significantly higher or lower than the norm? Both high costs and low costs can be a sign of problems in the development organization.
Having software inventory costs that are significantly higher than the numbers described here can be a symptom of long years of “technical debt,” which is a buildup of deferred essential work. It also can be a symptom of problems in the organizational structure of the development team or of the entire company.Often, it is one symptom of a weak or immature software development process. Other problems associated with an immature process include unpredictably late releases and a high number of product defects.
Weak development processes typically are seen in startup environments, where the focus is immediate deliverables instead of longer-term objectives. Weak processes also often are seen in “accidental software” companies, which typically are large hardware companies that have grown internal software development teams without enough exposure to industry best practices. These companies also tend to have executive management teams with deep hardware experience, but minimal software development experience. In these situations, executive management may be unable to provide proper oversight and guidance for software development.
Fortunately, there are many resources available for assessing and improving an organization’s development processes. One of the oldest and most widely used approaches is the Capability Maturity Model Integration developed by the Carnegie Mellon Software Engineering Institute.Another is ISO/IEC 15504, also known as SPICE (Software Process Improvement and Capability Determination).
Typically, an organization will begin the process improvement initiative with an appraisal or assessment from an outside body. This provides a baseline and gives the organization an indication of what process areas to focus on in order to improve. If an organization’s inventory costs are significantly lower than average, this may be a sign that a company is underinvesting and going into “technical debt.”
The technical debt metaphor was first introduced by Ward Cunningham in the 1990s to compare deficiencies in software with financial debt. There are many situations where a development team will decide to ship a product with known deficiencies in the software design and/or implementation.
Tactically, this can be the proper thing to do under certain circumstances. The application may be feature complete and functionally correct, while still having gone through shortcuts during development.
This is an example of technical debt in that the organization deliberately has underinvested in the development of the product in order to achieve a ship date or manage a development budget. Other examples of technical debt are product designs with insufficient architectures to support the overall product goals over time, and the design degradation of programs through poor feature implementation or program maintenance.
If a company is underinvesting in software development, as indicated either by developers consistently producing much more than 700 lines of logic code/month, or a very high number of overall lines of code, then that company may be accruing technical debt. Much like financial debt, technical debt eventually must be repaid, and also like financial debt, it compounds over time as developers work with increasingly fragile and compromised code. Ultimately, a company can work itself into a position of high development costs and low productivity as more of the organization’s resources are consumed paying interest on the debt, spending nearly all their time repairing the consequences of badly architected, poorly built, untested, or undocumented code.
Overinvestment and underinvestment in an organization’s software inventory may be symptomatic of problems in the development organization. This series of articles has presented tools for tracking a company’s software inventory and monitoring it over time. These tools can provide valuable insights into the overall health of the software development organization.
Wayne Lobb, Ph.D., is vice president and engineering director at Foliage, responsible for managing and overseeing consulting projects and outsourced software development projects for manufacturers of semiconductor equipment,industrial equipment and medical devices. David Warburton is the director of SystemsEngineering at Foliage. He has written numerous articles on medical device development and has more than 20 years of experience in the medical device and robotics industry managing technology-intensive, new-product development projects. Foliage is a full-service product development firm based in Burlington, Mass.