Performance testing is a class of tests implemented and executed to characterize and evaluate the performance-related
characteristics of the target-of-test, such as the timing profiles, execution flow, response times, and operational
reliability and limits. Different types of performance tests, each focused on a different test objective, are
implemented throughout the software development lifecycle (SDLC).
Early in the architecture iterations, performance tests are focused on identifying and eliminating
architectural-related performance bottlenecks. In the construction iterations, additional types of performance tests
are implemented and executed to fine-tune the software and environment (optimizing response time and resources), and to
verify that the applications and system acceptably handle high load and stress conditions, such as a large numbers of
transactions, clients, and/or volumes of data.
The following types of tests are included in Performance Testing:
-
Benchmark testing: Compares the performance of new or unknown target-of-test to a known reference standard,
such as existing software or measurements.
-
Contention test: Verifies the target-of-test can acceptably handle multiple actor demands on the same
resource (data records, memory, and so forth).
-
Performance profiling: Verifies the acceptability of the target-of-test's performance behavior using varying
configurations while the operational conditions remain constant.
-
Load testing: Verifies the acceptability of the target-of-test's performance behavior under varying
operational conditions (such as number of users, number of transactions, and so on) while the configuration remains
constant.
-
Stress testing: Verifies the acceptability of the target-of-test's performance behavior when abnormal or
extreme conditions are encountered, such as diminished resources or an extremely high number of users.
Performance evaluation is normally performed in conjunction with the User representative and is done from a
multileveled approach.
-
The first level of performance analysis involves evaluating the results for a single actor or use-case instance and
comparing the results across several test executions; for example, capturing the performance behavior of a single
actor performing a single use case without any other activity on the target-of-test and comparing the results with
several other test executions of the same actor or use case. This first-level analysis can help identify trends
that could indicate contention among system resources, which may affect the validity of the conclusions drawn from
other performance test results.
-
A second level of analysis examines the summary statistics and actual data values for specific actor or use-case
execution, and the target-of-test's performance behavior. Summary statistics include standard deviations and
percentile distributions for the response times, which provide an indication of the variability in system responses
as seen by individual actors.
-
A third level of analysis can help in understanding the causes and significance of performance problems. This
detailed analysis takes the low-level data and uses statistical methods to help testers draw correct conclusions
from the data. Detailed analysis provides objective and quantitative criteria for making decisions, but it's more
time consuming and requires a basic understanding of statistics.
Detailed analysis uses the concept of statistical significance to help understand when differences in
performance behavior are real or due to some random event associated with collecting the test data. The idea is that,
on a fundamental level, there is randomness associated with any event. Statistical testing determines whether there is
a systematic difference that can't be explained by random events.
See Concept: Key Measures of Test for more information on the different performance test
reports.
|