Teaching > CS777 - Software Reliability and Quality Assurance

DEPLOY

Software reliability is defined as the probability of failure-free operation of software for a specified time period within a given environment. This course motivates the need to engineer reliable software and focuses on approaches for measuring, assessing and improving reliability as part of a wider software quality assurance program. The course introduces the basic concepts of reliability (e.g., incidents, faults and failures), compares and contrasts software with hardware reliability, and examines a number of classic software reliability models. The course emphasizes techniques through which to assess the trustworthiness of reliability claims and predictions, and highlights a number of approaches to improve them. The course will also look at the role of measurement and testing in software reliability engineering, and explore the broader issue of software dependability (e.g., safety, availability and security). Software quality fundamentals and wider quality assurance activities will be examined throughout the course.

(c) United Feature Syndicate, Inc.

“Quality has to be caused, not controlled.” [Philip Crosby]

See here for Software Engineering material you should be familiar with.

Key Resources

• Lyu, Michael R. (Editor in Chief), Handbook of Software Reliability Engineering, McGraw-Hill, 1996.
  [available from: http://www.cse.cuhk.edu.hk/~lyu/book/reliability/]

Useful W eb Sites and Resources

Reliability:

• The Centre for Software Reliability (CSR), City University, London, UK
• Software Quality and Reliability Basics, John Musa
• Musa, John D., Software Reliability Engineering: More Reliable Software Faster and Cheaper, 2nd Edition, Author House, 2004 (background text - you do not need to buy)
• Pham, Hoang, Sofware Reliability, Springer-Verlag, 2000 (background text - you do not need to buy)
• Software Reliability Standards and Guidelines
• Software Tech E-News on Software Reliability
• NASA Overview of Software Reliability

PAPERS...

• Software Reliability and Dependability: A Roadmap, Bev Littlewood and Lorenzo Strigini (From "The Future of Software Engineering" Collection, 2000)
• Software Reliability Engineering: A Roadmap, Michael R. Lyu (From "The Future of Software Engineering" Collection, 2007) - slides here; paper found in blackboard
• Using Software Reliability Growth Models in Practice, Almering et al. (paper found in blackboard)
• New Ways to Get Accurate Software Reliability Measures, Sarah Brocklehurst and Bev Littlewood
• Recalibrating Software Reliability Models, Sarah Brocklehurst et al.
• Fundamental Concepts of Dependability, Avizienis et al.
• Dependability Assessment of Software-based Systems: State of the Art, Bev Littlewood
• The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software , Butler and Finelli; paper found in blackboard
• A nice exercise on Software Reliability Growth Modeling and Prediction by a student from Waterloo

Measurement and Metrics:

• Software Measurement: A Necessary Scientific Basis, Norman Fenton (IEEE TSE 1994), in Blackboard
• Status Report on Software Measurement, Pfleeger et al. (IEEE Software 1997), in Blackboard
• Sample Quality Characteristics and Metrics for Reusable Software (includes Reliability)
• Quality Control Charts
• The IT Metrics and Productivity Institute (excellent FREE webinars on topics related to this course)
• The Goal Question Metric Approach, Victor Basili et al. (see his contributions to Software Quality in blackboard too)
• GQM on Wikipedia
• Software Metrics Programs: setting up a useful program may be easier than you think, Norman Fenton
• A Software Metrics Primer, Karl Wiegers
• Metrics Traps to Avoid, Karl Wiegers (see blackboard)

Quality:

Learn About Some of the Quality Gurus...

• Philip Crosby (read bio synopsis)
• W. Edward Deming (read bio synopsis)
• Joseph M. Juran (read bio synopsis)

... and Some of the Quality Cults (look these up on Wikipedia too - good overviews provided there)

• Lean Manufacturing
• Lean Six Sigma (what the?)
• Kaizen (read the philosophy)
• Quality Circles (and their tools)
• Quality Function Deployment (House of Quality and Voice of the Customer)
• Six Sigma (read about Six Sigma straight from GE)
• Total Quality Management (TQM) - The Eight Elements of TQM (article) [Deming]
• Zero Defects / Right First Time [Crosby]

General Quality Starting Points

• Chapter 11 of the SWEBOK - "Software Quality"
• Compendium of SQA Resources
• The Quality Portal
• QA Links
• QA Forums
• Software Quality Assurance Resource Center
• Internet/Software Quality Hotlist
• Better Software and Sticky Minds
• Umpiring Quality - Article from Microsoft
• QA Guild - Quality Excellence

Your General SE Text Book's Quality Resources

• Sommerville’s Chapter on Quality Management
• Pressman’s Quality Management Resources

Quality Standards

• List of SQ Standards
• ISO9001 Definitions
• ISO9001 Checklist - ISO 9001 (SEPA 5/e, Chapter 8) is the standard that software organization must adopt if ISO Quality Certification is to be obtained
• ISO9003 (more specific to software, based on ISO9001)
• ISO 10011-1 1990 Audit Standard: Guidelines for Planning and Performing Quality Audits
• IEEE SQA Plan Standards (starting point)

Quality Requirements

• Quality CounterPoints by Gilb and Cockburn (paper found in blackboard)

Sample Guides, Plans and Checklists

• Project Quality Planning
• Sample QA Plan Checklist
• Sample SQA Plan
• SQA Plan Checklist
• SQA Plan Template from NASA
• Pressman’s Quality Review Checklist
• Dos and Don’ts in Establishing a Software Quality Plan for a Project
• On Assessing the SQA Process
• Change and Configuration Management and Control (Checklists and Forms)
• Sample Software Quality Test Report
• Quality Management System (QMS) Audits (Slides)
• NASA on Software Quality (one hopes they have learned and that this advice is sound!) Offers SQA Plans, Forms, Templates:
• A Tech Blogger’s Perspective on a Checklist for Managing Project Quality

Quality Certification

• Want to become a Certified Software Quality Engineer?
• American Society for Quality
• Quality Assurance Institute (USA)
• Body of Knowledge for a Software Quality Engineer -- outline of certification requirements

Quality of Life?

• I make this one up as I go along... there are no references for this... but I kind of think that THIS is living and THIS is living on the edge!

Testing:

• Your One Stop Guide to Testing
• Microsoft's Tester Center - Articles and Videos on Testing
• Anibal Sousa's Profile of a Great Software Tester

Code Review:

• Wiegers' Goodies for Peer Reviews
• Steve McConnell on Technical Reviews (Software Quality at Top Speed)
• Effective Code Reviews Without the Pain, Robert Bogue
• Peer Reviews, from the Encyclopedia of Software Engineering (Software Inspections and Walkthroughs)
• A Story About the Dos and Don'ts of Code Walkthroughs
• Code Reviews, Dr. Dobb's
• Macadamian's Code Review Checklist
• Five steps for performing an effective software product review
• Web link to a Mix of Studies (Pros / Cons of Code Reviews and Opportunity to Order a Free Book on the Topic: Best Kept Secrets of Peer Code Review)
• Globalization Step-by-Step: The World-Ready Approach to Testing and Localization Testing (Microsoft) -- more on Quality Assurance for World-Readiness at this Site
• Read up about Fagan Inspections (Original Paper - Design and code inspections to reduce errors in program development)

Process Improvement:

• Personal Software Process (PSP)
• Team Software Process (TSP)
• Capability Maturity Model Integration (CMMI)
• CMMI PPQA (Process and Product Quality Assurance) - Explore the CMU SEI Website
• Quality-based Process Improvement

Software Disasters (i.e., we do this reliability and quality stuff for a good reason!):

• KNOW THY RISKS - Risk Management (esp. the Team Aspects)
• The Risks Digest - More software bugs in this digest than roaches in an NYC apartment!
• 20 Famous Software Disasters
• The Top 10 IT Disasters of All Time
• History's Worst Software Bugs

Tools

• PETERS (Pre-Exploitative Tools for Evaluating Reliability of Software): Download tool here; PETERS website and tutorial; Usage instructions
• SMERFS3 (Statistical Modeling and Estimation of Reliability Functions for Software): Download tool here; SMERFS3 website and tutorial
• CASRE (Computer-Aided Software Reliability Estimation): Request download of tool here; CASRE User's Guide
• Quick link to Lyu's failure data set here

Weekly Notices

WEEK 1: 01/26/09 -> You have some preparatory tasks to do (task 1).

WEEK 2: 02/02/09 -> Work on task 2 - software forensics. Think about 'sufficient' reliability -- how would you determine it, specify it, measure it?

WEEK 3: 02/09/09 -> The forensics slides from this week are here. I included student slides where provided. Work on task 3 - it includes the movie watching. Take a look at the description for your first term paper -- please send me an email before next Monday to let me know what you would like to work on so I can guide you individually. Remember, no classes on Presidents' Day -- university is closed.

02/16/09 -> Presidents' Day - No Classes.

WEEK 4: 02/23/09 -> Work on task 4 -- there are some readings to do and your model research task. Please send me an abstract and proposed structure for your first term paper too -- or I will not be able to guide you individually in your research -- and you will be presenting this in class sooner than you realise!

03/02/09 -> SNOW DAY -- ALL PACE CLASSES CANCELLED. Please email me your work for task 4 and your term paper abstract/structure. Remind yourself of what is required and of the example papers. This week, research and draft your paper, prepare your presentation for Monday 9th March, and send me your working draft of the paper also for Monday 9th March for feedback. This is *individual* research and can be on anything related to software quality / reliability at all! If you are stuck, browse some of the links on this website and organise to see me this week!

WEEK 5: 03/09/09 -> Check the table of models created in class -- is it correct? is there anything missing? Time was allocated to hear about student papers -- work on these over the spring break as they are due on 03/23/09.

WEEK 6: 03/23/09 -> The analysis slides from the week are here. Please work on task 5 this week (all details in the task sheet).

WEEK 7: 03/23/09 -> Peer discussion - remember, you have the option to improve your term paper based on all your feedback... due next week with a cover sheet of improvements and new grade rationale! The analysis/tool slides from the week are here. Please try these things out for yourself -- as per task 6 -- and note that I'm interested in your systematic process and analystical thinking more than tooling expertise!

WEEK 8: 04/07/09 -> Work on task 7 and on putting it all together with a lifecycle process for Software Reliability Engineering and the Operational Profile. Watch out for term paper 2 ... it will be appearing soon, so be sure to be on top of your work.

WEEK 9: 04/13/09 -> Start thinking about term paper 2 ... it requires some planning and research. I will be posting some resources in blackboard to help with this task -- covering testing, planning a measurement and metrics gathering program (i.e., plan the tests and do something with the data), and quality planning guides/frameworks to help you (GQM being a simple example). Think about the full fault lifecycle, so other ways to increase reliability as discussed in class (process, standards, CMMI , design diversity, etc.) Be sure to read the handout: 'A Direct Path to Dependable Software' by Daniel Jackson (CACM April 2009) -- and think about how you could construct a case for yout term paper. I have a sample quality/dependability/reliability plan for you, but we will need to discuss that in class...so explore a bit first. [Remember to check out some of the links on this website too... and to find me a sample reliability requirement or failure intensity objective for next time...]

WEEK 10: 04/20/09 -> You had a guest from the PMI / ASQ who came to talk to you about quality tools this evening: Larry Cooke's presentation slides can be found here. Note, you may want to use a few of these tools to help you present ideas in your term paper. For those of you who missed the presentation, there are some additional paper handouts that Larry brought along -- but you will have to collect those from my office at some point since not in electronic form. Please work on planning/thinking about your term paper this week since you have to do some reading around. You do not have to do a presentation on this next week, but you should bring along some ideas on how you are going to be tackling this to the next class. We will look at metrics and dependability plans....

WEEK 11: 04/27/09 -> Read up on Software Measurement and Metrics (there are a number of links above and it is recommended you read Chapter 1 and 2 from the Fenton text (available in Blackboard) ... skim the other papers on the topic in there too). Then, also refresh yourself with GQM (links above and resources in blackboard). Think about the data requirement and data collection task for incident reporting so as to gather the data you need to do reliability analysis... see if you could design a form and suitable database schema based on the questions you need answers to. How would you collect and process this data? We skimmed a sample of part of a dependability plan and accompanying material is in a zip folder in blackboard -- use this as part of a guide for your work (but note the need to explain/justify any targets you state and cover other aspects in your planning too -- the zip folder with SQA resources in blackboard may help but you are meant to draw upon everything we have covered and that you have read, and to synthesise this material).

Week 12: 05/04/09 -> Students presented their term paper work - focusing on the quality analysis and the structure of the plan -- what is to be included in it and why. This should be sufficient for a customer to gain a high-level idea as to how they are going to get a quality product and for a development team to understand what they are going to need to do to engineer and assure one. The ISO9126 Quality Model characteristics we discussed are listed here. Please be sure to re-read key articles and refresh yourself on your task work etc. prior to the final exam.

TERM PAPER IS DUE BEFORE MIDNIGHT EST ON SUNDAY MAY 10 , 2009 -- email me your work and I will respond when I receive it. If you have no response from me come midnight, send it again. Do not zip your work on the spam filter will block it. No late work is acceptable given this is a 3 day extension. Anything late will start a full letter grade lower per 12 hours overdue...i.e, the max you can get is a B if within 12 hours late, then a C if within 24 hours late! FINAL EXAM 05/11/09 5pm-8pm. Closed book. No make-ups. Be there!

05/13/09 grades emailed to Pace accounts

Last updated 05/13/09