Measuring Quality

The measurement of Quality, whether Product or Process, requires the collection and analysis of information, usually stated in terms of measurements and metrics. Measurements are made primarily to gain control of a project, and therefore be able to manage it. They are also used to evaluate how close or far we are from the objectives set in the plan in terms of completion, quality, compliance to requirements, etc.

Metrics are used to attain two goals, knowledge and change (or achievement):

Knowledge goals: they are expressed by the use of verbs like evaluate, predict, monitor. You want to better understand your development process. For example, you may want to assess product quality, obtain data to predict testing effort, monitor test coverage, or track requirements changes.

Change or achievement goals: these are expressed by the use of verbs such as increase, reduce, improve, or achieve. You are usually interested in seeing how things change or improve over time, from an iteration to another, from a project to another.

Metrics for both goals are used for measuring Process and Product Quality.

All metrics require criteria to identify and to determine the degree or level at which of acceptable quality is attained. The level of acceptable quality is negotiable and variable, and needs to be determined and agreed upon early in the development lifecycle For example, in the early iterations, a high number of application defects are acceptable, but not architectural ones. In late iterations, only aesthetic defects are acceptable in the application.

The acceptance criteria may be stated in many ways and may include more than one measure. Common acceptance criteria may include the following measures:

• Defect counts and / or trends, such as the number of defects identified, fixed, or that remain open (not fixed).
• Test coverage, such as the percentage of code, or use cases planned or implemented and executed (by a test). Test coverage is usually used in conjunction with the defect criteria identified above).
• Performance, such as the time required for a specified action (use case, operation, or other event) to occur. This is criteria is commonly used for Performance testing, Failover and recovery testing, or other tests in which time criticality is essential.
• Compliance. This criteria indicates the degree to which each work product, activity, task, or step must meet an agreed upon standard or guideline.
• Acceptability or satisfaction. This criteria is usually used with subjective measures, such as usability or aesthetics.

See Concept: Metrics for additional information.

Measuring Product Quality

Stating the requirements in a clear, concise, and testable fashion is only part of achieving product quality. It is also necessary to identify the measures and criteria that will be used to identify the desired level of quality and determine if it has been achieved. Measures describe the method used to capture the data used to assess quality, while criteria defines the level or point at which the product has achieved acceptable (or unacceptable) quality.

Measuring the product quality of an executable work product is achieved using one or more measurement techniques, such as:

• reviews / walkthroughs
• inspection
• execution

Different metrics are used, dependent upon the nature the quality goal of the measure. For example, in reviews, walkthroughs, and inspections, the primary goal is to focus on the function and reliability quality dimensions. Defects, coverage, and compliance are the primary metrics used when these measurement techniques are used. Execution however, may focus on function, reliability, or performance. Therefore defects, coverage, and performance are the primary metrics used. Other measures and metrics will vary based upon the nature of the requirement.

See Concept: Key Measures of Test for additional information.

See  Guideline: Metrics for additional information.

See Concept: Product Quality for additional information.

Measuring Process Quality

The measurement of Process Quality is achieved by collecting both knowledge and achievement measures.

1. The degree of adherence to the standards, guidelines, and implementation of an accepted process.
2. Status / state of current process implementation to planned implementation.
3. The quality of the work products created (using product quality measures described above).

Measuring process quality is achieved using one or more measurement techniques, such as:

• progress - such as use cases demonstrated or milestones completed
• variance - differences between planned and actual schedules, budgets, staffing requirements, etc.
• product quality measures and metrics (as described in Measuring Product Quality section above)

See  Guideline: Metrics for additional information.

See Introduction to Project Management for additional information.

See Concept: Process Quality for additional information.