Cisco Systems, Inc.
Development Defect Density – A Software Process Effectiveness Metric
Cisco Systems is currently developing, implementing, and using several types of software process effectiveness metrics, including Development Defect Density (Dev DD), which is used primarily to focus on the quality of code exiting the ‘development’ phase (i.e., the lifecycle processes upstream from function test). Several business units have developed and are currently using defect density variants, and an initiative is also underway to roll out standardized defect density across the company (along with several supporting metrics and guidance on how to interpret and use the density data).
This paper describes our experience with Dev DD and its variants, including comparisons and correlations with other effectiveness metrics, and the relationship between density trends and the customer experience. Some of our findings are:
1. Dev DD consistently is a strong predictor of customer-found defects and post-release internal finds. In prediction models for 13 software releases, Dev DD is the dominant variable in 7, while variants are essential in another 3.
2. Dev DD levels correlate well (adj. r-squared=0.84, P=0.02) with development escapes identified and confirmed by engineers and testers. This consistency both validates the density numbers and enables us to exploit the particular strengths of the density view.
3. Goaling Dev DD can be useful, as long as several ancillary metrics (such as post-development percent customer-found defects) are provided to make sure that function and system test effectiveness are not substantially changing.
One of the goals of our corporate rollout is to identify process and data management changes that will not only improve the accuracy and availability of the density data, but also indicate which additional development processes should be standardized. The individual business unit and corporate implementations have been difficult, and this paper will describe the specific problems encountered.