Software quality

Software quality is motivated by at least two main perspectives:

• Risk management: Software failure has caused more than inconvenience. Software errors can cause human fatalities (see for example: List of software bugs). The causes have ranged from poorly designed user interfaces to direct programming errors,{{Cite web|last=Clark|first=Mitchell|date=2021-02-24|title=iRobot says it'll be a few weeks until it can clean up its latest Roomba software update mess|url=https://www.theverge.com/2021/2/24/22299346/irobot-roomba-update-issues-vacuums-fix-several-weeks|access-date=2021-02-25|website=The Verge|language=en}}{{Cite web|title=Top 25 Software Errors |url=https://www.sans.org/top25-software-errors/|access-date=2021-02-25|website=www.sans.org}}{{Cite web|title='Turn it Off and On Again Every 149 Hours' Is a Concerning Remedy for a $300 Million Airbus Plane's Software Bug|url=https://gizmodo.com/turn-it-off-and-on-again-every-149-hours-is-a-concernin-1836818094|access-date=2021-02-25|website=Gizmodo|date=30 July 2019 |language=en-us}} see for example Boeing 737 case or Unintended acceleration cases{{Cite web|title=MISRA C, Toyota and the Death of Task X|url=http://blog.gimpel.com/2015/02/misra-c-toyota-and-death-of-task-x.html|access-date=2021-02-25}}{{Cite web|title=An Update on Toyota and Unintended Acceleration « Barr Code|url=https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/|access-date=2021-02-25|website=embeddedgurus.com}} or Therac-25 cases.[http://sunnyday.mit.edu/papers/therac.pdf Medical Devices: The Therac-25*] {{webarchive|url=https://web.archive.org/web/20080216083852/http://sunnyday.mit.edu/papers/therac.pdf |date=2008-02-16 }}, Nancy Leveson, University of Washington This resulted in requirements for the development of some types of software, particularly and historically for software embedded in medical and other devices that regulate critical infrastructures: "[Engineers who write embedded software] see Java programs stalling for one third of a second to perform garbage collection and update the user interface, and they envision airplanes falling out of the sky.".[http://ptolemy.eecs.berkeley.edu/publications/papers/02/embsoft/embsoftwre.pdf Embedded Software] {{webarchive|url=https://web.archive.org/web/20100705084849/http://ptolemy.eecs.berkeley.edu/publications/papers/02/embsoft/embsoftwre.pdf |date=2010-07-05 }}, Edward A. Lee, To appear in Advances in Computers

(Marvin Victor Zelkowitz, editor), Vol. 56, Academic Press, London, 2002, Revised from UCB ERL Memorandum M01/26

University of California, Berkeley, CA 94720, USA, November 1, 2001 In the United States, within the Federal Aviation Administration (FAA), the FAA Aircraft Certification Service provides software programs, policy, guidance and training, focus on software and Complex Electronic Hardware that has an effect on the airborne product (a "product" is an aircraft, an engine, or a propeller).{{cite web|url=http://www.faa.gov/aircraft/air_cert/design_approvals/air_software|title=Aircraft Certification Software and Airborne Electronic Hardware|access-date=28 September 2014|url-status=live|archive-url=https://web.archive.org/web/20141004152626/http://www.faa.gov/aircraft/air_cert/design_approvals/air_software/|archive-date=4 October 2014}} Certification standards such as DO-178C, ISO 26262, IEC 62304, etc. provide guidance.

• Cost management: As in any other fields of engineering, a software product or service governed by good software quality costs less to maintain, is easier to understand and can change more cost-effective in response to pressing business needs.{{Cite web|title=The Cost of Poor Software Quality in the US: A 2020 Report {{!}} CISQ - Consortium for Information & Software Quality|url=https://www.it-cisq.org/the-cost-of-poor-software-quality-in-the-us-a-2020-report.htm|access-date=2021-02-25|website=www.it-cisq.org}} Industry data demonstrate that poor application structural quality in core business applications (such as enterprise resource planning (ERP), customer relationship management (CRM) or large transaction processing systems in financial services) results in cost, schedule overruns and creates waste in the form of rework (see Muda (Japanese term)).{{Cite web|title=What is Waste? {{!}} Agile Alliance|url=https://www.agilealliance.org/what-is-waste/|access-date=2021-02-25|website=Agile Alliance {{!}}|date=20 April 2016 |language=en-US}}{{Cite web|date=January 26, 2018|first=Scott|last=Matteson|title=Report: Software failure caused $1.7 trillion in financial losses in 2017|url=https://www.techrepublic.com/article/report-software-failure-caused-1-7-trillion-in-financial-losses-in-2017/|access-date=2021-02-25|website=TechRepublic|language=en}}{{Cite web|last=Cohane|first=Ryan|date=2017-11-16|title=Financial Cost of Software Bugs|url=https://medium.com/@ryancohane/financial-cost-of-software-bugs-51b4d193f107|access-date=2021-02-25|website=Medium|language=en}} Moreover, poor structural quality is strongly correlated with high-impact business disruptions due to corrupted data, application outages, security breaches, and performance problems.{{Citation|last1=Eloff|first1=Jan|title=Software Failures: An Overview|date=2018|url=http://link.springer.com/10.1007/978-3-319-61334-5_2|work=Software Failure Investigation|pages=7–24|place=Cham|publisher=Springer International Publishing|language=en|doi=10.1007/978-3-319-61334-5_2|isbn=978-3-319-61333-8|access-date=2021-02-25|last2=Bella|first2=Madeleine Bihina}}
• CISQ reports on the cost of poor quality estimates an impact of:
• $2.08 trillion in 2020{{Cite web|title=Poor software quality cost businesses $2 trillion last year and put security at risk|url=https://www.ciodive.com/news/poor-software-quality-report-2020/593015/|access-date=2021-02-26|website=CIO Dive|language=en-US}}{{Cite web|title=Synopsys-Sponsored CISQ Research Estimates Cost of Poor Software Quality in the US $2.08 Trillion in 2020|url=https://finance.yahoo.com/news/synopsys-sponsored-cisq-research-estimates-140500352.html|access-date=2021-02-26|website=finance.yahoo.com|language=en-US}}
• [https://www.it-cisq.org/the-cost-of-poor-quality-software-in-the-us-a-2018-report/The-Cost-of-Poor-Quality-Software-in-the-US-2018-Report.pdf $2.84 trillion in 2018]
• IBM's Cost of a Data Breach Report 2020 estimates that the average global costs of a data breach:{{Cite web|date=2020-08-06|title=What Does a Data Breach Cost in 2020?|url=https://digitalguardian.com/blog/what-does-data-breach-cost-2020|access-date=2021-03-08|website=Digital Guardian}}{{Cite web|date=2020|title=Cost of a Data Breach Report 2020 {{!}} IBM|url=https://www.ibm.com/security/digital-assets/cost-data-breach-report/|access-date=2021-03-08|website=www.ibm.com}}
• $3.86 million

Software quality is the "capability of a software product to conform to requirements."{{Cite web|title=ISO - ISO 9000 family — Quality management|url=https://www.iso.org/iso-9001-quality-management.html|access-date=2021-02-24|website=ISO|language=en}}{{Cite web|title=ISO/IEC/IEEE 24765:2017|url=https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/07/19/71952.html|access-date=2021-02-24|website=ISO|language=en}} while for others it can be synonymous with customer- or value-creation{{Cite web|title=Mastering automotive software |url=https://www.mckinsey.com/industries/automotive-and-assembly/our-insights/when-code-is-king-mastering-automotive-software-excellence|access-date=2021-02-25|website=www.mckinsey.com}}{{Cite web|title=ISO/IEC 25010:2011|url=https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/03/57/35733.html|access-date=2021-02-24|website=ISO|language=en}} or even defect level.{{Cite book|last=Wallace|first=D.R.|title=Proceedings 26th Annual NASA Goddard Software Engineering Workshop |chapter=Practical software reliability modeling |date=2002|chapter-url=https://ieeexplore.ieee.org/document/992668|location=Greenbelt, MD, USA|publisher=IEEE Comput. Soc|pages=147–155|doi=10.1109/SEW.2001.992668|isbn=978-0-7695-1456-7|s2cid=57382117}} Software quality measurements can be split into three parts: process quality, product quality which includes internal and external properties and lastly, quality in use, which is the effect of the software.{{Cite web |title=ISO/IEC 25023:2016 |url=https://www.iso.org/standard/35747.html |access-date=2023-11-06 |website=ISO |language=en}}

ASQ uses the following definition: Software quality describes the desirable attributes of software products. There are two main approaches exist: defect management and quality attributes.{{Cite web|title=What is Software Quality? {{!}} ASQ|url=https://asq.org/quality-resources/software-quality|access-date=2021-02-24|website=asq.org}}

Software Assurance (SA) covers both the property and the process to achieve it:{{Cite journal|title=SAMATE - Software Assurance Metrics And Tool Evaluation project main page|url=https://samate.nist.gov/Main_Page.html|access-date=2021-02-26|journal=NIST|date=3 February 2021}}

• [Justifiable] confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at any time during its life cycle and that the software functions in the intended manner
• The planned and systematic set of activities that ensure that software life cycle processes and products conform to requirements, standards, and procedures

The Project Management Institute's PMBOK Guide "Software Extension" defines not "Software quality" itself, but Software Quality Assurance (SQA) as "a continuous process that audits other software processes to ensure that those processes are being followed (includes for example a software quality management plan)." whereas Software Quality Control (SCQ) means "taking care of applying methods, tools, techniques to ensure satisfaction of the work products toward quality requirements for a software under development or modification."{{Cite book|url=https://www.worldcat.org/oclc/959513383|title=Software extension to the PMBOK guide|date=2013|others=Project Management Institute|isbn=978-1-62825-041-1|edition=5th|location=Newtown Square, Pennsylvania|oclc=959513383}}

The first definition of quality in recorded history is from Shewhart in the beginning of 20th century: "There are two common aspects of quality: one of them has to do with the consideration of the quality of a thing as an objective reality independent of the existence of man. The other has to do with what we think, feel or sense as a result of the objective reality. In other words, there is a subjective side of quality."{{Cite book|last=Shewart|first=Walter A.|url=https://www.worldcat.org/oclc/1108913766|title=Economioc control of quality of manufactured product.|date=2015|publisher=Martino Fine Books|isbn=978-1-61427-811-5|location=[Place of publication not identified]|oclc=1108913766}}

Kitchenham and Pfleeger, further reporting the teachings of David Garvin, identify five different perspectives on quality:{{Cite journal|last1=Kitchenham|first1=B.|author1-link=Barbara Kitchenham|last2=Pfleeger|first2=S. L.|date=January 1996|title=Software quality: the elusive target [special issues section]|url=https://ieeexplore.ieee.org/document/476281|journal=IEEE Software|volume=13|issue=1|pages=12–21|doi=10.1109/52.476281|issn=1937-4194}}{{Cite book|last=Garvin|first=David A.|url=https://www.worldcat.org/oclc/16005388|title=Managing quality : the strategic and competitive edge|date=1988|publisher=Free Press|isbn=0-02-911380-6|location=New York|oclc=16005388}}

• The transcendental perspective deals with the metaphysical aspect of quality. In this view of quality, it is "something toward which we strive as an ideal, but may never implement completely".B. Kitchenham and S. Pfleeger, "Software quality: the elusive target", IEEE Software, vol. 13, no. 1, pp. 12–21, 1996. It can hardly be defined, but is similar to what a federal judge once commented about obscenity: "I know it when I see it".{{Cite book|last=Kan|first=Stephen H.|url=https://www.worldcat.org/oclc/50149641|title=Metrics and models in software quality engineering|date=2003|publisher=Addison-Wesley|isbn=0-201-72915-6|edition=2nd|location=Boston|oclc=50149641}}
• The user perspective is concerned with the appropriateness of the product for a given context of use. Whereas the transcendental view is ethereal, the user view is more concrete, grounded in the product characteristics that meet user's needs.
• The manufacturing perspective represents quality as conformance to requirements. This aspect of quality is stressed by standards such as ISO 9001, which defines quality as "the degree to which a set of inherent characteristics fulfills requirements" (ISO/IEC 9001International Organization for Standardization, "ISO/IEC 9001: Quality management systems -- Requirements," 1999.).
• The product perspective implies that quality can be appreciated by measuring the inherent characteristics of the product.
• The final perspective of quality is value-based. This perspective recognizes that the different perspectives of quality may have different importance, or value, to various stakeholders.

{{Blockquote|text=The problem inherent in attempts to define the quality of a product, almost any product, was stated by the master Walter A. Shewhart. The difficulty in defining quality is to translate the future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay. This is not easy, and as soon as one feels fairly successful in the endeavor, he finds that the needs of the consumer have changed, competitors have moved in, etc.W. E. Deming, "Out of the crisis: quality, productivity and competitive position". Cambridge University Press, 1988.|author=W. Edwards Deming}}

{{Blockquote|Quality is a customer determination, not an engineer's determination, not a marketing determination, nor a general management determination. It is based on the customer's actual experience with the product or service, measured against his or her requirements -- stated or unstated, conscious or merely sensed, technically operational or entirely subjective -- and always representing a moving target in a competitive market.A. V. Feigenbaum, "Total Quality Control", McGraw-Hill, 1983.}}

{{Blockquote|The word quality has multiple meanings. Two of these meanings dominate the use of the word: 1. Quality consists of those product features which meet the need of customers and thereby provide product satisfaction. 2. Quality consists of freedom from deficiencies. Nevertheless, in a handbook such as this it is convenient to standardize on a short definition of the word quality as "fitness for use".J.M. Juran, "Juran's Quality Control Handbook", McGraw-Hill, 1988.}}

Tom DeMarco has proposed that "a product's quality is a function of how much it changes the world for the better."{{Citation needed|date=February 2021}} This can be interpreted as meaning that functional quality and user satisfaction are more important than structural quality in determining software quality.

Another definition, coined by Gerald Weinberg in Quality Software Management: Systems Thinking, is "Quality is value to some person."{{Cite book|last=Weinberg|first=Gerald M.|url=https://www.worldcat.org/oclc/23870230|title=Quality software management: Volume 1, Systems Thinking|publisher=Dorset House|year=1991|isbn=0-932633-22-6|location=New York, N.Y.|oclc=23870230}}{{Cite book|last=Weinberg|first=Gerald M.|url=https://www.worldcat.org/oclc/23870230|title=Quality software management: Volume 2, First-Order Measurement|publisher=Dorset House|year=1993|isbn=0-932633-22-6|location=New York, N.Y.|oclc=23870230}}

One of the challenges in defining quality is that "everyone feels they understand it"Crosby, P., Quality is Free, McGraw-Hill, 1979 and other definitions of software quality could be based on extending the various descriptions of the concept of quality used in business.

Software quality also often gets mixed-up with Quality Assurance

or Problem Resolution Management{{Cite web|title=SUP.9 – Problem Resolution Management - Kugler Maag Cie|url=https://www.kuglermaag.com/automotive-spice/sup9-problem-resolution-management/|access-date=2021-02-25|website=www.kuglermaag.com}}

or Quality Control{{Cite web|last=Hoipt|date=2019-11-29|title=Organizations often use the terms 'Quality Assurance' (QA) vs 'Quality Control' (QC)…|url=https://medium.com/truemoney-engineering/organizations-often-use-the-terms-quality-assurance-qa-vs-quality-control-qc-24f673d32d6e|access-date=2021-02-25|website=Medium|language=en}}

or DevOps.

It does overlap with these areas (see also PMI definitions), but it is distinctive as it does not solely focus on testing but also on processes, management, improvements, assessments, etc.

{{Anchor|SoftwareQualityMeasurement}}

Although the concepts presented in this section are applicable to both structural and functional software quality, measurement of the latter is essentially performed through software testing.{{Cite journal|last1=Wallace|first1=D.|last2=Watson|first2=A. H.|last3=Mccabe|first3=T. J.|date=1996-08-01|title=Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric|journal=NIST |url=https://www.nist.gov/publications/structured-testing-testing-methodology-using-cyclomatic-complexity-metric|language=en}} Testing is not enough: According to one study, "individual programmers are less than 50% efficient at finding bugs in their own software. And most forms of testing are only 35% efficient. This makes it difficult to determine [software] quality."{{Cite web|last=Bellairs|first=Richard|title=What Is Code Quality? And How to Improve Code Quality|url=https://www.perforce.com/blog/sca/what-code-quality-and-how-improve-code-quality|access-date=2021-02-28|website=Perforce Software|language=en}}

File:SoftwareQualityCharacteristicAttributeRelationship.png

Software quality measurement is about quantifying to what extent a system or software possesses desirable characteristics. This can be performed through qualitative or quantitative means or a mix of both. In both cases, for each desirable characteristic, there are a set of measurable attributes the existence of which in a piece of software or system tend to be correlated and associated with this characteristic. For example, an attribute associated with portability is the number of target-dependent statements in a program. More precisely, using the Quality Function Deployment approach, these measurable attributes are the "hows" that need to be enforced to enable the "whats" in the Software Quality definition above.

The structure, classification and terminology of attributes and metrics applicable to software quality management have been derived or extracted from the ISO 9126-3 and the subsequent ISO/IEC 25000:2005 quality model. The main focus is on internal structural quality. Subcategories have been created to handle specific areas like business application architecture and technical characteristics such as data access and manipulation or the notion of transactions.

The dependence tree between software quality characteristics and their measurable attributes is represented in the diagram on the right, where each of the 5 characteristics that matter for the user (right) or owner of the business system depends on measurable attributes (left):

• Application Architecture Practices
• Coding Practices
• Application Complexity
• Documentation
• Portability
• Technical and Functional Volume

Correlations between programming errors and production defects unveil that basic code errors account for 92 percent of the total errors in the source code. These numerous code-level issues eventually count for only 10 percent of the defects in production. Bad software engineering practices at the architecture levels account for only 8 percent of total defects, but consume over half the effort spent on fixing problems, and lead to 90 percent of the serious reliability, security, and efficiency issues in production.{{Cite web|title=OMG Whitepaper {{!}} CISQ - Consortium for Information & Software Quality|url=https://www.it-cisq.org/omg-cisq-wp/index.htm|access-date=2021-02-26|website=www.it-cisq.org}}{{cite web |url=http://www.omg.org/CISQ_compliant_IT_Systemsv.4-3.pdf |title=How to Deliver Resilient, Secure, Efficient and Agile IT Systems in Line with CISQ Recommendations - Whitepaper | Object Management Group |access-date=2013-10-18 |url-status=live |archive-url=https://web.archive.org/web/20131228132152/http://www.omg.org/CISQ_compliant_IT_Systemsv.4-3.pdf |archive-date=2013-12-28 }}

Many of the existing software measures count structural elements of the application that result from parsing the source code for such individual instructions{{Cite web|title=Software Size Measurement: A Framework for Counting Source Statements|url=https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=11689|access-date=2021-02-24|website=resources.sei.cmu.edu| date=31 August 1992 |language=en}} tokens{{Cite book|last=Halstead|first=Maurice H.|url=https://dl.acm.org/doi/book/10.5555/540137|title=Elements of Software Science (Operating and programming systems series)|date=1977|publisher=Elsevier Science Inc.|isbn=978-0-444-00205-1|location=USA|doi=}} control structures (Complexity), and objects.{{Cite journal|last1=Chidamber|first1=S. R.|last2=Kemerer|first2=C. F.|date=June 1994|title=A metrics suite for object oriented design|url=https://ieeexplore.ieee.org/document/295895|journal=IEEE Transactions on Software Engineering|volume=20|issue=6|pages=476–493|doi=10.1109/32.295895|issn=1939-3520|hdl=1721.1/48424|s2cid=9493847 |hdl-access=free}}

Software quality measurement is about quantifying to what extent a system or software rates along these dimensions. The analysis can be performed using a qualitative or quantitative approach or a mix of both to provide an aggregate view [using for example weighted average(s) that reflect relative importance between the factors being measured].

This view of software quality on a linear continuum has to be supplemented by the identification of discrete Critical Programming Errors. These vulnerabilities may not fail a test case, but they are the result of bad practices that under specific circumstances can lead to catastrophic outages, performance degradations, security breaches, corrupted data, and myriad other problems{{Cite book|last=Nygard|first=Michael|url=https://www.worldcat.org/oclc/1102387436|title=Release It!|date=2007|others=an O'Reilly Media Company Safari|publisher=Pragmatic Bookshelf |isbn=978-0978739218|edition=1st|oclc=1102387436}} that make a given system de facto unsuitable for use regardless of its rating based on aggregated measurements. A well-known example of vulnerability is the Common Weakness Enumeration,{{Cite web|url=https://cwe.mitre.org/|title=CWE - Common Weakness Enumeration|website=cwe.mitre.org|access-date=2016-05-20|url-status=live|archive-url=https://web.archive.org/web/20160510002103/http://cwe.mitre.org/|archive-date=2016-05-10}} a repository of vulnerabilities in the source code that make applications exposed to security breaches.

The measurement of critical application characteristics involves measuring structural attributes of the application's architecture, coding, and in-line documentation, as displayed in the picture above. Thus, each characteristic is affected by attributes at numerous levels of abstraction in the application and all of which must be included in calculating the characteristic's measure if it is to be a valuable predictor of quality outcomes that affect the business. The layered approach to calculating characteristic measures displayed in the figure above was first proposed by Boehm and his colleagues at TRW (Boehm, 1978)Boehm, B., Brown, J.R., Kaspar, H., Lipow, M., MacLeod, G.J., & Merritt, M.J. (1978). Characteristics of Software Quality. North-Holland. and is the approach taken in the ISO 9126 and 25000 series standards. These attributes can be measured from the parsed results of a static analysis of the application source code. Even dynamic characteristics of applications such as reliability and performance efficiency have their causal roots in the static structure of the application.

Structural quality analysis and measurement is performed through the analysis of the source code, the architecture, software framework, database schema in relationship to principles and standards that together define the conceptual and logical architecture of a system. This is distinct from the basic, local, component-level code analysis typically performed by development tools which are mostly concerned with implementation considerations and are crucial during debugging and testing activities.

The root causes of poor reliability are found in a combination of non-compliance with good architectural and coding practices. This non-compliance can be detected by measuring the static quality attributes of an application. Assessing the static attributes underlying an application's reliability provides an estimate of the level of business risk and the likelihood of potential application failures and defects the application will experience when placed in operation.

Assessing reliability requires checks of at least the following software engineering best practices and technical attributes:

{{Div col}}

• Application Architecture Practices
• Coding Practices
• Complexity of algorithms
• Complexity of programming practices
• Compliance with Object-Oriented and Structured Programming best practices (when applicable)
• Component or pattern re-use ratio
• Dirty programming
• Error & Exception handling (for all layers - GUI, Logic & Data)
• Multi-layer design compliance
• Resource bounds management
• Software avoids patterns that will lead to unexpected behaviors
• Software manages data integrity and consistency
• Transaction complexity level

{{Div col end}}

Depending on the application architecture and the third-party components used (such as external libraries or frameworks), custom checks should be defined along the lines drawn by the above list of best practices to ensure a better assessment of the reliability of the delivered software.

As with Reliability, the causes of performance inefficiency are often found in violations of good architectural and coding practice which can be detected by measuring the static quality attributes of an application. These static attributes predict potential operational performance bottlenecks and future scalability problems, especially for applications requiring high execution speed for handling complex algorithms or huge volumes of data.

Assessing performance efficiency requires checking at least the following software engineering best practices and technical attributes:

• Application Architecture Practices
• Appropriate interactions with expensive and/or remote resources
• Data access performance and data management
• Memory, network and disk space management
• Compliance with Coding Practices (Best coding practices)

Software quality includes software security.{{Cite web|date=2019-05-24|title=Code quality and code security: How are they related? {{!}} Synopsys|url=https://www.synopsys.com/blogs/software-security/code-quality-code-security/|access-date=2021-03-09|website=Software Integrity Blog|language=en-US}} Many security vulnerabilities result from poor coding and architectural practices such as SQL injection or cross-site scripting.{{Cite web|date=2020|title=Cost of a Data Breach Report 2020 {{!}} IBM|url=https://www.ibm.com/security/digital-assets/cost-data-breach-report/|access-date=2021-03-09|website=www.ibm.com}}{{Cite web|date=2020-08-27|title=Key Takeaways from the 2020 Cost of a Data Breach Report|url=https://www.bluefin.com/bluefin-news/key-takeaways-ibm-ponemons-2020-cost-data-breach-report/|access-date=2021-03-09|website=Bluefin|language=en-US}} These are well documented in lists maintained by CWE,{{cite web |url=http://cwe.mitre.org/ |title=CWE - Common Weakness Enumeration |publisher=Cwe.mitre.org |access-date=2013-10-18 |url-status=live |archive-url=https://web.archive.org/web/20131014120515/http://cwe.mitre.org/ |archive-date=2013-10-14 }} and the SEI/Computer Emergency Center (CERT) at Carnegie Mellon University.{{Cite web|title=SEI CERT Coding Standards - CERT Secure Coding - Confluence|url=https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards|access-date=2021-02-24|website=wiki.sei.cmu.edu}}

Assessing security requires at least checking the following software engineering best practices and technical attributes:

• Implementation, Management of a security-aware and hardening development process, e.g. Security Development Lifecycle (Microsoft) or IBM's Secure Engineering Framework.{{Cite book|url=http://www.redbooks.ibm.com/abstracts/redp4641.html?Open|title=Security in Development: The IBM Secure Engineering Framework {{!}} IBM Redbooks|date=2016-09-30|language=en-US}}
• Secure Application Architecture Practices{{Cite book|url=http://www.redbooks.ibm.com/abstracts/sg246014.html?Open|title=Enterprise Security Architecture Using IBM Tivoli Security Solutions {{!}} IBM Redbooks|date=2016-09-30|language=en-US}}{{Cite web|title=Secure Architecture Design Definitions {{!}} CISA|url=https://us-cert.cisa.gov/ics/Secure-Architecture-Design-Definitions|access-date=2021-03-09|website=us-cert.cisa.gov}}
• Multi-layer design compliance
• Security best practices (Input Validation, SQL Injection, Cross-Site Scripting, Access control etc.){{Cite web|title=OWASP Foundation {{!}} Open Source Foundation for Application Security|url=https://owasp.org/|access-date=2021-02-24|website=owasp.org|language=en}}{{cite web|url=http://www.sans.org/top25-programming-errors/ |title=CWE's Top 25 |publisher=Sans.org |access-date=2013-10-18}}
• Secure and good Programming Practices
• Error & Exception handling

Maintainability includes concepts of modularity, understandability, changeability, testability, reusability, and transferability from one development team to another. These do not take the form of critical issues at the code level. Rather, poor maintainability is typically the result of thousands of minor violations with best practices in documentation, complexity avoidance strategy, and basic programming practices that make the difference between clean and easy-to-read code vs. unorganized and difficult-to-read code.[http://www.ifsq.org/resources/level-2/booklet.pdf IfSQ Level-2 A Foundation-Level Standard for Computer Program Source Code] {{webarchive|url=https://web.archive.org/web/20111027000121/http://www.ifsq.org/resources/level-2/booklet.pdf |date=2011-10-27 }}, Second Edition August 2008, Graham Bolton, Stuart Johnston, IfSQ, Institute for Software Quality.

Assessing maintainability requires checking the following software engineering best practices and technical attributes:

{{Div col}}

• Application Architecture Practices
• Architecture, Programs and Code documentation embedded in source code
• Code readability
• Code smells
• Complexity level of transactions
• Complexity of algorithms
• Complexity of programming practices
• Compliance with Object-Oriented and Structured Programming best practices (when applicable)
• Component or pattern re-use ratio
• Controlled level of dynamic coding
• Coupling ratio
• Dirty programming
• Documentation
• Hardware, OS, middleware, software components and database independence
• Multi-layer design compliance
• Portability
• Programming Practices (code level)
• Reduced duplicate code and functions
• Source code file organization cleanliness

{{Div col end}}

Maintainability is closely related to Ward Cunningham's concept of technical debt, which is an expression of the costs resulting of a lack of maintainability. Reasons for why maintainability is low can be classified as reckless vs. prudent and deliberate vs. inadvertent,{{cite web|url=http://martinfowler.com/bliki/TechnicalDebtQuadrant.html|title=TechnicalDebtQuadrant|last1=Fowler|first1=Martin|date=October 14, 2009|access-date=February 4, 2013|url-status=live|archive-url=https://web.archive.org/web/20130202111651/http://martinfowler.com/bliki/TechnicalDebtQuadrant.html|archive-date=February 2, 2013}}{{cite web | title=Code quality: a concern for businesses, bottom lines, and empathetic programmers | website=Stack Overflow | date=2021-10-18 | url=https://stackoverflow.blog/2021/10/18/code-quality-a-concern-for-businesses-bottom-lines-and-empathetic-programmers/ | ref={{sfnref | Stack Overflow | 2021}} | access-date=2023-12-05}} and often have their origin in developers' inability, lack of time and goals, their carelessness and discrepancies in the creation cost of and benefits from documentation and, in particular, maintainable source code.{{cite book |chapter=Architectural design and documentation: Waste in agile development?|last1=Prause|first1=Christian|title=2012 International Conference on Software and System Process (ICSSP)|pages=130–134|last2=Durdik|first2=Zoya|date=June 3, 2012|publisher=IEEE Computer Society|doi=10.1109/ICSSP.2012.6225956|isbn=978-1-4673-2352-9|s2cid=15216552|url=http://publica.fraunhofer.de/documents/N-216042.html }}

Measuring software size requires that the whole source code be correctly gathered, including database structure scripts, data manipulation source code, component headers, configuration files etc. There are essentially two types of software sizes to be measured, the technical size (footprint) and the functional size:

• There are several software technical sizing methods that have been widely described. The most common technical sizing method is number of Lines of Code (#LOC) per technology, number of files, functions, classes, tables, etc., from which backfiring Function Points can be computed;
• The most common for measuring functional size is function point analysis. Function point analysis measures the size of the software deliverable from a user's perspective. Function point sizing is done based on user requirements and provides an accurate representation of both size for the developer/estimator and value (functionality to be delivered) and reflects the business functionality being delivered to the customer. The method includes the identification and weighting of user recognizable inputs, outputs and data stores. The size value is then available for use in conjunction with numerous measures to quantify and to evaluate software delivery and performance (development cost per function point; delivered defects per function point; function points per staff month.).

The function point analysis sizing standard is supported by the International Function Point Users Group (IFPUG). It can be applied early in the software development life-cycle and it is not dependent on lines of code like the somewhat inaccurate Backfiring method. The method is technology agnostic and can be used for comparative analysis across organizations and across industries.

Since the inception of Function Point Analysis, several variations have evolved and the family of functional sizing techniques has broadened to include such sizing measures as COSMIC, NESMA, Use Case Points, FP Lite, Early and Quick FPs, and most recently Story Points. Function Point has a history of statistical accuracy, and has been used as a common unit of work measurement in numerous application development management (ADM) or outsourcing engagements, serving as the "currency" by which services are delivered and performance is measured.

One common limitation to the Function Point methodology is that it is a manual process and therefore it can be labor-intensive and costly in large scale initiatives such as application development or outsourcing engagements. This negative aspect of applying the methodology may be what motivated industry IT leaders to form the Consortium for IT Software Quality focused on introducing a computable metrics standard for automating the measuring of software size while the IFPUG keep promoting a manual approach as most of its activity rely on FP counters certifications.

CISQ defines Sizing as to estimate the size of software to support cost estimating, progress tracking or other related software project management activities. Two standards are used: Automated Function Points to measure the functional size of software and Automated Enhancement Points to measure the size of both functional and non-functional code in one measure.{{Cite web|title=Software Sizing Standards {{!}} CISQ - Consortium for Information & Software Quality|url=https://www.it-cisq.org/standards/software-sizing-standards/|access-date=2021-01-28|website=www.it-cisq.org}}

{{Anchor|CriticalProgrammingErrors}}

Critical Programming Errors are specific architectural and/or coding bad practices that result in the highest, immediate or long term, business disruption risk.{{Cite web|title=Why Software fails|url=https://spectrum.ieee.org/why-software-fails|access-date=2021-03-20|website=IEEE Spectrum: Technology, Engineering, and Science News|date=2 September 2005|language=en}}

These are quite often technology-related and depend heavily on the context, business objectives and risks. Some may consider respect for naming conventions while others – those preparing the ground for a knowledge transfer for example – will consider it as absolutely critical.

Critical Programming Errors can also be classified per CISQ Characteristics. Basic example below:

• Reliability
• Avoid software patterns that will lead to unexpected behavior (Uninitialized variable, null pointers, etc.)
• Methods, procedures and functions doing Insert, Update, Delete, Create Table or Select must include error management
• Multi-thread functions should be made thread safe, for instance servlets or struts action classes must not have instance/non-final static fields
• Efficiency
• Ensure centralization of client requests (incoming and data) to reduce network traffic
• Avoid SQL queries that don't use an index against large tables in a loop
• Security
• Avoid fields in servlet classes that are not final static
• Avoid data access without including error management
• Check control return codes and implement error handling mechanisms
• Ensure input validation to avoid cross-site scripting flaws or SQL injections flaws
• Maintainability
• Deep inheritance trees and nesting should be avoided to improve comprehensibility
• Modules should be loosely coupled (fanout, intermediaries) to avoid propagation of modifications
• Enforce homogeneous naming conventions

Newer proposals for quality models such as Squale and Quamoco{{Cite journal|title = Operationalised product quality models and assessment: The Quamoco approach|url = http://elib.uni-stuttgart.de/opus/volltexte/2015/9916/pdf/quamoco.pdf|journal = Information and Software Technology|date = 2015|pages = 101–123|volume = 62|doi = 10.1016/j.infsof.2015.02.009|first1 = Stefan|last1 = Wagner|first2 = Andreas|last2 = Goeb|first3 = Lars|last3 = Heinemann|first4 = Michael|last4 = Kläs|first5 = Constanza|last5 = Lampasona|first6 = Klaus|last6 = Lochmann|first7 = Alois|last7 = Mayr|first8 = Reinhold|last8 = Plösch|first9 = Andreas|last9 = Seidl|arxiv = 1611.09230|s2cid = 10992384}} propagate a direct integration of the definition of quality attributes and measurement. By breaking down quality attributes or even defining additional layers, the complex, abstract quality attributes (such as reliability or maintainability) become more manageable and measurable. Those quality models have been applied in industrial contexts but have not received widespread adoption.

{{Div col}}

• Accessibility
• Availability
• Best coding practices
• Coding conventions
• Cohesion and Coupling
• Computer bug
• Cyclomatic complexity
• Defect criticality
• Dependability
• GQM
• ISO/IEC 9126
• Software Process Improvement and Capability Determination - ISO/IEC 15504
• Programming style
• Quality: quality control, total quality management.
• Requirements management
• Scope (project management)
• Security
• Security engineering
• Software architecture
• Software bug
• Software quality assurance
• Software quality control
• Software metrics
• Software reusability
• Software standard
• Software testing
• Static program analysis
• Testability

{{Div col end}}

== Further reading ==

• Android OS [https://developer.android.com/quality Quality Guidelines] including checklists for UI, Security, etc. July 2021
• Association of Maritime Managers in Information Technology & Communications (AMMITEC). [https://www.ammitec.org/images/AMMITEC_Maritime_Software_Quality_Guidelines_v1.pdf Maritime Software Quality Guidelines]. September 2017
• Capers Jones and Olivier Bonsignour, "The Economics of Software Quality", Addison-Wesley Professional, 1st edition, December 31, 2011, {{ISBN|978-0-13-258220-9}}
• [https://cnescatlab.github.io/ CAT Lab - CNES Code Analysis Tools Laboratory] (on GitHub)
• Girish Suryanarayana, Software Process versus Design Quality: Tug of War?{{Cite journal|last1=Suryanarayana|first1=Girish|year=2015|title=Software Process versus Design Quality: Tug of War?|journal=IEEE Software|volume=32|issue=4|pages=7–11|doi=10.1109/MS.2015.87|s2cid=9226051|doi-access=free}}
• Ho-Won Jung, Seung-Gweon Kim, and Chang-Sin Chung. [http://doi.ieeecomputersociety.org/10.1109/MS.2004.1331309 Measuring software product quality: A survey of ISO/IEC 9126]. IEEE Software, 21(5):10–13, September/October 2004.
• International Organization for Standardization. Software Engineering—Product Quality—Part 1: Quality Model. ISO, Geneva, Switzerland, 2001. ISO/IEC 9126-1:2001(E).
• [http://www.sei.cmu.edu/library/abstracts/presentations/esepg.cfm Measuring Software Product Quality: the ISO 25000 Series and CMMI (SEI site)]
• [http://arxiv-web3.library.cornell.edu/abs/1408.3253 MSQF - A measurement based software quality framework] Cornell University Library
• Omar Alshathry, Helge Janicke, "Optimizing Software Quality Assurance," compsacw, pp. 87–92, 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops, 2010.
• Robert L. Glass. Building Quality Software. Prentice Hall, Upper Saddle River, NJ, 1992.
• Roland Petrasch, "[https://www.researchgate.net/publication/258105237_The_Definition_of_Software_Quality-_A_Practical_Approach The Definition of 'Software Quality': A Practical Approach]", ISSRE, 1999
• Software Quality Professional,{{Cite web|title=Software Quality Professional {{!}} ASQ|url=https://asq.org/quality-resources/pub/software-quality-professional|access-date=2021-01-28|website=asq.org}} American Society for Quality (ASQ)
• Software Quality Journal{{Cite web|title=Software Quality Journal|url=https://www.springer.com/journal/11219|access-date=2021-01-28|website=Springer|language=en}} by Springer Nature
• {{cite book

| last1 = Spinellis | first1 = Diomidis

| title = Code quality: the open source perspective

| date = 2006-04-04

| publisher = Addison-Wesley Professional

| location = Upper Saddle River, New Jersey, US

| isbn = 978-0-321-16607-4

| url = https://www.spinellis.gr/codequality/

}}

• Stephen H. Kan. Metrics and Models in Software Quality Engineering. Addison-Wesley, Boston, MA, second edition, 2002.
• Stefan Wagner. [https://www.springer.com/us/book/9783642385704 Software Product Quality Control]. Springer, 2013.

Notes

{{reflist|colwidth=30em}}

Bibliography

{{refbegin}}

• {{citation |last=Albrecht |first=A. J.|title=Measuring application development productivity. In Proceedings of the Joint SHARE/GUIDE IBM Applications Development Symposium.|publisher=IBM|year=1979}}
• {{citation |last1=Ben-Menachem|first1= M.|last2=Marliss|first2=G. S.|year=1997|title=Software Quality, Producing Practical and Consistent Software|publisher=Thomson Computer Press}}
• {{citation |last1=Boehm|first1= B.|last2=Brown|first2=J.R.|last3=Kaspar|first3=H.|last4=Lipow|first4=M.|last5=MacLeod|first5=G.J.|last6=Merritt|first6=M.J.|year=1978| title=Characteristics of Software Quality|publisher=North-Holland.}}
• {{citation |last1=Chidamber|first1= S.|last2=Kemerer|first2=C.|year=1994|title=A Metrics Suite for Object Oriented Design. IEEE Transactions on Software Engineering, 20 (6)|pages=476–493}}
• {{citation |last1=Ebert|first1=Christof|last2=Dumke|first2=Reiner|title=Software Measurement: Establish - Extract - Evaluate - Execute|publisher=Kindle Edition|page=91}}
• {{citation |last1=Garmus|first1=D.|last2=Herron|first2=D.|year=2001| title =Function Point Analysis|publisher=Addison Wesley}}
• {{citation |last=Halstead|first=M.E.|year=1977|title=Elements of Software Science|publisher=Elsevier North-Holland}}
• {{citation |last1=Hamill|first1=M.|last2=Goseva-Popstojanova|first2=K.|year=2009|title=Common faults in software fault and failure data. IEEE Transactions of Software Engineering, 35 (4)|pages=484–496}}
• {{citation |last=Jackson|first=D.J.|year=2009|title=A direct path to dependable software. Communications of the ACM, 52 (4).}}
• {{citation |last=Martin|first=R.|year=2001|title=Managing vulnerabilities in networked systems|publisher=IEEE Computer.}}
• {{citation |last=McCabe|first=T.|title=A complexity measure. IEEE Transactions on Software Engineering|date=December 1976}}
• {{citation |last=McConnell |first=Steve |title=Code Complete |edition=First |publisher=Microsoft Press |year=1993}}
• {{citation |last=Nygard|first=M.T.|year=2007|title=Release It! Design and Deploy Production Ready Software|publisher=The Pragmatic Programmers.}}
• {{citation |last=Park|first=R.E.|year=1992|title=Software Size Measurement: A Framework for Counting Source Statements. (CMU/SEI-92-TR-020). |publisher=Software Engineering Institute, Carnegie Mellon University}}
• {{cite book |last=Pressman |first=Roger S. |author-link=Roger S. Pressman |year=2005 |title=Software Engineering: A Practitioner's Approach |edition=Sixth International |publisher=McGraw-Hill Education |isbn=0071267824}}
• {{citation |last=Spinellis|first=D.|year=2006|title=Code Quality|publisher=Addison Wesley}}

{{refend}}

{{Commons category}}

• [https://www.mckinsey.com/industries/automotive-and-assembly/our-insights/when-code-is-king-mastering-automotive-software-excellence When code is king: Mastering automotive software excellence] (McKinsey, 2021)
• [https://users.ece.cmu.edu/~koopman/lectures/2016_issre.pdf Embedded System Software Quality: Why is it so often terrible? What can we do about it?] (by [https://users.ece.cmu.edu/~koopman/ Philip Koopman])
• [https://www.it-cisq.org/standards/code-quality-standards/ Code Quality Standards] by CISQ™
• CISQ Blog: https://blog.it-cisq.org
• [http://microelectronics.esa.int/vhdl/pss/PSS-05-11.pdf Guide to software quality assurance] (ESA)
• [http://emits.sso.esa.int/emits-doc/e_support/Bssc962.pdf Guide to applying the ESA software engineering standards to small software projects] (ESA)
• [https://www.nasa.gov/centers/ivv/pdf/192949main_ESA_Garcia.pdf An Overview of ESA Software Product Assurance Services] (NASA/ESA)
• [https://lsoares.medium.com/our-approach-to-quality-in-volkswagen-software-dev-center-lisbon-72f5728e2235 Our approach to quality in Volkswagen Software Dev Center Lisbon]
• [https://google.github.io/styleguide/ Google Style Guides]
• [https://www.infoq.com/news/2011/03/Ensuring-Product-Quality-Google/ Ensuring Product Quality at Google] (2011)
• [https://sma.nasa.gov/sma-disciplines/software-assurance NASA Software Assurance]
• [https://www.nist.gov/itl/ssd/software-quality-group NIST Software Quality Group]
• OMG/CISQ [https://www.it-cisq.org/standards/automated-function-points/ Automated Function Points] (ISO/IEC 19515)
• [https://www.omg.org/spec/ATDM/ OMG Automated Technical Debt Standard]
• [https://re-magazine.ireb.org/articles/automated-quality-assurance Automated Quality Assurance] (articled in IREB by Harry Sneed)
• [https://www.nist.gov/publications/structured-testing-testing-methodology-using-cyclomatic-complexity-metric Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric] (1996)
• [https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2015/code-quality/analyzing-application-quality-by-using-code-analysis-tools?view=vs-2015 Analyzing Application Quality by Using Code Analysis Tools] (Microsoft, Documentation, Visual Studio, 2016)

{{Software quality}}

{{Software engineering}}

{{Computer science}}

{{Authority control}}

Category:Systems thinking

Quality

Category:Source code