Software Risk Workshop

Tuesday November 11, 2008

The 'Software Risk Workshop' focuses on the software risk management discipline, which attempts to 'identify, address, and eliminate software risk items before they become either threats to successful software operation or major sources of software rework' (Boehm, 1989). The two components of risk management are risk assessment (identification, analysis, and prioritization) and risk control (planning, resolution, and monitoring).

    The goals for the software risk workshop are:
  • to devote time to all these facets of risk management in a holistic way
  • to discuss how risk management practice can be leveraged in today's real world environment
  • to share the latest techniques and methods to identify and mitigate reliability and security risks.

The workshop will be highly interactive, with several panel discussions and focused breakout sessions:

Initial morning panel: The topic 'What are the key indicators of software risk?' will be discussed briefly by a panel of industry practitioners and academic researchers. Discussion: What are the metrics and indicators that enable us to assess the level of risk, and how do we use this information to enhance software engineering processes in order to reduce and mitigate risk?
Panel members are: Steve Kan (IBM), Dave Catlett (Microsoft), Michael Gegick (N.C. State U.), Bob Mullen (Cisco), and Pete Rotella (Cisco), with Rick Follenweider (Cisco) as moderator.

Morning breakout sessions: Four breakout teams will address the main topic, 'What are the key indicators of software risk?,' each from a particular perspective:

Models/metrics - Can we construct models to accurately predict risk? What types of models are effective, and what are their individual strengths and limitations? How do we validate models for individual releases?
Breakout Facilitator: Pete Rotella (Cisco).

System-level issues - How should we adjust risk prediction/computation to adjust for fault tolerance constraints? How do we include backward compatibility considerations, documentation errors, system dependencies, etc., in risk planning, prediction, and management?
Breakout Facilitator: Dave Catlett (Microsoft).

Security - Can we predict security vulnerabilities? How do we know when to target specific software components for more testing, inspections, and re-design?
Breakout Facilitator: Michael Gegick (N.C. State U.).

Business/Process - Can we exploit development processes to reduce risk? Are there accurate ways to estimate business risk (e.g., risk to revenue, profit, ROI)? Can we predict the risk to the customer, from a business perspective. How do we integrate producer risk and customer risk in risk planning?
Breakout Facilitator: Bob Mullen (Cisco).

Morning wrap-up panel: Output from the four breakout sessions will be discussed by the panel members and all breakout team members. Ideas and experience will be shared, and a wrapup summary report will be produced for the participants.

Afternoon discussion topic: The topic 'Can we reduce risk by continual risk management /computation early in (and throughout) the lifecycle?' will be introduced by Rick Follenweider (Cisco). Can we, from the beginning to the end of the lifecycle, plan for and compute risk iteratively? As the software development project progresses, can we effectively focus on reducing the variance, the risk?

Afternoon presentations: Five individual 15-minute presentations will focus on the major afternoon topic from the perspectives of the four breakout areas mentioned above - models/metrics, system-level issues, security, mitigation. Our goal is to have these presentations also represent industry, government, and academic perspectives. Following each presentation will be 15 minutes of discussion. The presentations are:

  • Steve Kan (IBM) - A QPI (Quality Predictive Indicators) Process for Risk and Quality Management for Software Projects
  • Dave Catlett (Microsoft) - Windows Engineering Risk Management Tools
  • Sunita Chulani (Cisco) - A Risk Management Approach to Project Management
  • Rick Follenweider (Cisco) - Defect Discovery Rate (DDR) - Managing Risk During Testing
  • Michael Gegick (N.C. State U.) and Pete Rotella (Cisco) - Predicting Software Security Vulnerabilities.

Afternoon wrap-up panel: The panel members, the presenters, and breakout team participants will discuss the 'software lifecycle' topic, and a wrapup summary report of these discussions will be produced for the participants.

Workshop wrap-up: Conclusions, future workshop ideas, attendees' feedback.

Panel members and presenters:
  • Dave Catlett - Principal Test Architect, Windows Engineering Tools, Microsoft. Worked at all levels of software testing for past 18 years. Develop risk evaluation tools and systems focused on helping developers, testers, and project decision makers make data-driven decisions based on a quantitative understanding of the risk associated with code software. Authored "Applying Software Risk Models: Shipping Windows Vista" at ISSRE 2006; and The Practical Guide to Defect Prevention, Microsoft Press, authored Chapter 6: Improving the Testability of Software and Chapter 8: Risk Analysis.
  • Bob Mullen - Technical Leader, Corporate Quality Metrics, Cisco Systems, Inc. Currently measuring reliability of successive versions of software (ISSRE-2008). Presented a Lognormal SW Reliability Growth Model (ISSRE-1998) and rolled out Orthogonal Defect Classification at Cisco (ISSRE-2000). Developed models of code coverage growth, defect frequencies, and defect repair times with Swapna Gokhale, and co-authored "Applications of the Lognormal to Software Reliability Engineering" in Handbook of Performability Engineering, ed. K.B. Misra, Springer, 2008.
  • Richard Follenweider -Advisory Engineer, Cisco Systems, Inc. Rick has worked for Cisco for 8 years, most recently in the Release Operations area. Prior to Cisco, he worked at Storage Technology Corporation, a computer peripherals company, performing analyses of a highly fault-tolerant disk system called Iceberg, which used RAID 7+ architecture. His interest includes the prediction/measurement of fielded reliability/ availability, the use of release readiness tools and presenting at ISSRE.
  • Dr. Stephen H. Kan -Senior Technical Staff Member at IBM Rochester, Minnesota. Steve is the cross-series software quality strategy lead for IBM's Systems and Technology Group. Prior to his current role, he was the technical manager of software quality for IBM's System i for many years, covering all aspects of quality ranging from quality goal setting, supplier quality requirements, quality plans, in-process metrics, statistical process control and quality assessments, to defect modeling, reliability projections, field quality tracking, and customer satisfaction. He is the author of the book Metrics and Models in Software Quality Engineering, numerous technical reports, and articles and chapters in professional journals. He is experienced in project and process assessments, including leading an organization in achieving CMM Level 5 assessment. He is a member of the American Society for Quality (ASQ) since 1989. He is certified by ASQ as a Quality Engineer (CQE), a Reliability Engineer (CRE), and a Manager of Quality/Organizational Excellence (CMQ/OE), and by the Project Management Institute as a Project Management Professional (PMP). Dr. Kan is an adjunct faculty member of the University of Minnesota Master of Science in Software Engineering (MSSE) program since 1998.
  • Sunita Chulani - recently joined Cisco; responsible for software data analytics. Prior to joining Cisco, Dr. Chulani was a research staff member at IBM's T.J. Watson Research Center. Her interests include the entire software engineering life cycle, particularly measurement and analysis. She received her PhD in Computer Science from the University of Southern California. She has several patents and has authored numerous journal papers, conference papers, book chapters and has co-authored Software Cost Estimation with COCOMO II.
  • Pete Rotella -focuses primarily on software metrics at Cisco. Recent projects include implementation of processes and metrics for development phase containment, escape detection, and defect density. Also active in developing models that use in-process metrics to predict downstream behavior, including internally-found defects, customer-found defects, and security vulnerabilities.

Contact: Pete Rotella [protella AT cisco DOT com] or Rick Follenweider (rfollenw AT cisco DOT com)
Less Info...