Article
09/09/2009

Quality-Attribute Auditing: The What, Why, and How

Hui Zhu

May 2007

Summary: Software quality is a key factor in the long-term success of any company. Quality software will increase profitability, reduce maintenance cost, and achieve higher customer satisfaction and loyalty. Software-quality audits will help ensure that the software development achieves a high level of quality. (6 printed pages)

A few lines of code can wreak more havoc than a bomb.

–Tom Ridge, (Former) Secretary, U.S. Department of Homeland Security

Introduction
Auditing Reveals the Problems
What Is Software Quality?
Conduct Software-Quality Audits Throughout the Software-Development Life Cycle
Testing Is the Key Process to Ensure Software Quality
Quality Requires Ongoing Commitment from All Parties
Conclusion
Lessons Learned and Takeaways
Critical-Thinking Questions
Sources
Further Reading
Glossary

Introduction

Jim received the phone call with alarm. "Customers are calling to complain that their credit cards are being charged fraudulently after visiting our Web site. What is up with this?"

As part of their business expansion, Contoso, Ltd, had hired Jim as a software architect to develop a Web site that would allow customers to purchase CDs online. So far, so good. Three months went by without a problem, and, after $100,000 in sales, Contoso declared the site a huge success. In fact, the management team was planning a big marketing campaign to promote the site—until this happened.

Jim was confounded. How could this have happened? He went back over the Web server log and, to his dismay, discovered the culprit: A hacker had exploited a SQL injection vulnerability in the CD search page, which gave him unfettered access to customers' personal data, as well as credit-card information in the database.

The Web site was taken down immediately. Senior management was shocked with the incident. The development team stopped the current projects and spent three days implementing input validation to fix the vulnerability before putting it back online.

However, senior management had lost confidence in the quality of the software. An IT auditor was brought in to investigate the problem and find a solution, to ensure the future quality of the site.

Auditing Reveals the Problems

After reviewing the software-development processes and software artifacts, the IT auditor discovered a few problems and produced the findings to the management team. The findings were scathing. There were many things that Jim had overlooked when designing the site:

Jim had not identified and documented detailed security requirements in the software-requirements specifications.
Jim's development team did not take into account input-validation security requirements during the design and development. In other words, he had not sanitized the user input.
Jim's current quality audit had focused on the software-development process only; security testing had been ignored, because of the tight deadline and lack of team skills in this area.

Jim was at a loss. However, in talking with other software architects, he learned that this was an opportunity to improve his skills. Indeed, he had to improve, to win back the confidence of the management team.

Unfortunately, Contoso was not the only company with software-quality problems. In fact, a lapse in attention to quality has been a common problem for software-development projects worldwide. However, the quality of Web site security features turned out to be a key factor in the long-term success of Jim's company. He realized that improving the quality of his software would increase customer satisfaction and loyalty, as well as enhance profitability and return on investments.

Jim started his learning journey to improve software quality along the development processes and prevent such an incident from happening again.

What Is Software Quality?

In Webster's dictionary, quality is defined as: "The essential character of something, an inherent or distinguishing character, degree, or grade of excellence." Software quality must be defined in terms that are meaningful to the software's users. What is most important to them, and what do they need?

Software quality is commonly described in terms that are known as quality attributes. A quality attribute is a property of a software product that will be judged directly by stakeholders. Quality attributes are—and should be—quantifiable in specifications by appropriate and practical scales of measure. The ISO software-quality model [ISO 2001] defines six quality-attribute categories: functionality, reliability, usability, efficiency, maintainability, and portability. The categories, in turn, are further subdivided into subcharacteristics (see Figure 1).

Figure 1. Quality model for internal and external quality (Click on the picture for a larger image)

To be useful, software-quality attributes need not only be defined, but also confirmed and enforced. The IEEE [IEEE 1997] defines a software-quality audit as:

An independent examination of a software product, software process, or set of software processes to assess compliance with specifications, standards, contractual agreements, or other criteria. It provides an independent evaluation regarding conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures.

Conduct Software-Quality Audits Throughout the Software-Development Life Cycle

Jim realized that the detailed and measurable quality attributes (security) that he needed were not explicitly identified and documented in the software-requirements specification. He and the developers built the software strictly based on that specification; consequently, security was overlooked in the design and development of the software. If he had implemented a rigorous quality-audit program from the earliest phase of his software-development life cycle, he could have detected his problems earlier and avoided the costly and embarrassing situation that his company faced. Although better late than never, Jim did not begin his software-quality audit until after software development had been completed, the Web site had been deployed, and a major problem had occurred.

Software quality is best achieved during the early stages, when the cost to remedy defects is one to two orders of magnitude less than it would be during testing or post-production maintenance. It is, therefore, advantageous to find and correct defects as near to their point of origin as possible. Thus, a quality audit should happen as early as possible in the software-development life cycle. Generally, however, a software-development life cycle is composed of many phases, including inception, requirement, design, development, testing, and deployment. Errors can occur at any point in this process. Therefore, a rigorous quality-audit program should be a continuous process that is executed in parallel with the entire software-development life cycle.

Formal quality audits, performed at the end of each development phase, verify that the products of that phase meet requirements with respect to software-quality attributes. Auditors review, test, and challenge deliverables against quality requirements and are responsible for delivering a report that specifies problems that development teams will be held responsible for resolving.

Informal audits can be performed in parallel with software-development activities by the architect and the developers themselves. Informal audits can give quick feedback on quality issues by the people who are familiar with the software artifacts, which makes the quality of formal audits more effective and efficient.

Testing Is the Key Process to Ensure Software Quality

Good software-development processes alone are not able to guarantee the quality of software. Software defects can be introduced during any phase of the software-development life cycle, despite rigorous processes. Traditional quality audits that focus on processes compliance do not provide adequate assurance on the quality of software.

Contoso had quality-assurance processes in place to ensure compliance with their software-development methodology. However, their document reviews did not help to uncover the damaging SQL injection vulnerabilities that were eventually exploited by hackers and thieves. Quality (security) testing had been largely ignored, because of tight deadlines and lack of security skills. A skilled security tester definitely would have detected the SQL injection vulnerability prior to production release and prevented the costly incident in production.

The goals of testing are to find defects and verify that the software meets its requirements—including quality attributes, as perceived by the user. It is unfortunate that, in many cases, testing programs are actually aimed at functionality validation only. This falls short of the real goals of a sound testing program. Testing that does not cover quality attributes generally is not considered complete.

Incorporating software-quality audits into testing programs provides a reasonable assurance of the software quality. All quality attributes specified in the software-requirements specification should be tested. Only through thorough testing can an auditor gain a certain level of confidence in the quality of the software under review.

Quality Requires Ongoing Commitment from All Parties

Quality does not happen by chance. The process must be managed. All persons involved in a software-development project must be held responsible for meeting their quality-attribute requirements. The process also must be communicated and understood. Quality audits are held not only to find software defects or nonconformance issues, but also to promote the software-quality understanding in the development team. Before quality auditing or testing, the approach of the audit team and the techniques that they will use should be communicated to management, architects, and developers, so that everyone understands what the audit will entail and how deficiencies might be detected. In addition to helping the audit process proceed efficiently, this also provides everyone an opportunity to think through ways of improving quality throughout development. As soon as they are armed with this knowledge, all participants can become "auditors" who can voluntarily "audit" their own work at any time.

Conclusion

Ultimately, Jim's company, Contoso, was able to bounce back from its security breach; Contoso was lucky enough to catch its software-quality issues in time to prevent further damage. Although damage to the company's fortunes was minimal, the incident cost Jim credibility in the short term. However, the incident did teach Jim the valuable lesson that all this grief could have been prevented if only he had carried out software-quality audits and software-quality testing early in the development cycle. His mistake was one that he would never make again.

Ultimately, Jim became a better software architect from this experience.

Lessons Learned and Takeaways

Software quality is a key factor in the long-term success of any company. Quality software will increase profitability, reduce maintenance cost, and achieve higher customer satisfaction and loyalty. Software-quality audits will help ensure that the software development achieves a high level of quality.

However, it is not trivial to perform software-quality audits in a complete and effective manner. Software-quality audits should be carried out throughout the software-development life cycle, to verify and validate artifacts that are produced at the end of each phase. Effective audits should not only focus on process conformance, but also incorporate software testing into each phase to improve confidence in the degree of software quality.

A quality audit is a tool that promotes quality in development teams. Only when everyone involved in a development project understands the goals and techniques of quality assurance and quality audits can they begin to act on that knowledge. As soon as that happens, a high degree of quality of software can be achieved.

Critical-Thinking Questions

What is a software-quality attribute?
How can we ensure the quality of software?
How can we audit the software quality in each phase of software-development project?
What must be audited in a software-development project?

Sources

[IEEE 1997] IEEE Computer Society, Software Engineering Technical Committee. 1997. IEEE Standard for Software Reviews: Description. (IEEE Std 1028-1997.) New York, NY: IEEE, 1998. [Cited 2007 January 10]
[ISO 2001] International Organization for Standardization, International Electrotechnical Commission. Software Engineering: Product Quality: Part 1: Quality Model. (ISO/IEC 9126-1:2001.) Geneva, Switzerland: ISO/IEC, 2001. [Cited 2007 January 10]

Glossary

SQL injection—A hacking technique that attempts to pass SQL commands through a Web application for execution by the back-end database. If not sanitized properly, Web applications might result in SQL injection attacks that allow hackers to view, modify, and delete data from the database. It is one of the many Web attack mechanisms that is used by hackers to steal data from organizations.

About the author

Hui Zhu is a consultant with over 10 years of extensive experience in security-architecture design and implementation, security-management system, and security audit and review in varied IT environments. Leveraging expertise in both security management and security consulting, Hui has established a proven approach for various security-architecture practices, and has successfully helped clients to manage risk and achieve their security objectives. You can contact Hui at zhusec@gmail.com.

This article was published in Skyscrapr, an online resource provided by Microsoft. To learn more about architecture and the architectural perspective, please visit skyscrapr.net.