Basics of Performance Testing

Concepts and Background

• Software testing is the process of executing software and comparing the observed behavior to the desired behavior. The major goal of software testing is to discover errors in the software, with a secondary goal of building confidence in the proper operation of the software when testing does not discover errors.
• Testing alone does not produce software quality. Testing ensures that the software functions as it was initially requested by those who dictate the requirements. Therefore, not testing software increases the risk that the software may not comply with the requirements and won't necessarily provide the expected added business value.
• In the literature and lexicon of software testing, the terms load testing, performance testing, scalability testing, and stress testing have many (sometimes overlapping) meanings. For the purpose of this document, the basic definition of load testing is: "Testing that involves executing operations simultaneously, at the same rates and with the same other environmental conditions as those that will occur in the field; thus the same interactions and impact of environmental conditions will occur as can be expected in the field."

Goal of Performance Testing

• Within the discipline of software testing, the goal of performance testing is to exercise the software and hardware components of a system under varying conditions of load in order to discover errors (bottlenecks, defects, single-points-of-failure, and resource limitations) and to build confidence (responsiveness, robustness, reliability, efficiency, scalability, and resiliency).
• Performance testing is based on the practice of modeling the expected usage of a software program or system by simulating multiple users accessing the system's services concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as a web application. However, software systems can be load-tested in other ways also. For example, a system can be forced to transmit or process an extremely large document; or a system can be forced to generate a report based on several years' worth of data.
• Performance testing proves that the system-under-test meets (or does not meet) its performance requirements or objectives, specifically those regarding response time, throughput, and utilization under a given (usually, but not necessarily, average or normal) load. Therefore, performance testing activities should seek to measure and evaluate response times, transaction rates, and other load and time sensitive requirements of an application in order to verify that required performance has been achieved.
• When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high loads or projected growth levels, it is known as stress testing. The load is usually so great that error conditions are the expected result, although the system should stabilize at levels less than design limits. Stress testing submits the SUT to extremes of workload processing and resource utilization. The general approach is to drive one or more components or the entire system to processing, utilization, or capacity limits by submitting as much workload as is necessary (all at once or increasing incrementally) to achieve that end, even forcing it to its breaking point. The purpose of stress testing is to reveal defects in software that only appear, or are more easily noticed, when the system is stressed to and beyond its limits.
• Background testing is the execution of normal functional testing while the SUT is exercised by a realistic work load. This work load is being processed "in the background" as far as the functional testing is concerned. Background testing differs from normal functional testing in that another test driver submits a constant "background" workload to the SUT for the duration of the functional tests. The background workload can be an average or moderate number of simulated users, actively exercising the system. The rationale for doing background testing is to test the system's functionality and responsiveness under more realistic conditions than single-user or simulated single-user functional testing allows.

Minimal Entrance Criteria for Performance Testing

• Performance requirements must be finalized and testable, must have passed inspection and must have been signed off by the Business Analyst, Quality Analyst, and Performance Test Engineer.
• The Performance Test Approach must be finalized, must have passed inspection and must have been signed off by the Business Analyst, Quality Analyst, and Performance Test Engineer.
• The performance testing timeline must be reflective of test development, execution, and analysis/reporting time estimates and resource constraints provided by the Performance Test Engineer.
• All developed code must be unit and system tested and free of critical defects. Code development must be completed and signed off as a load-ready release by the development team.
• All test hardware and environments must be in place, must have passed integration testing, and must be scheduled for exclusive use during performance tests.
• Test hardware, environments, and data-loading must be production-like to ensure accurate performance analysis. If a production-like test environment is not feasible, performance testing may commence if the Business Analyst, Quality Analyst, and the Performance Test Engineer are in full agreement that the risk is manageable.
• All code and configuration changes must be frozen. If a code or configuration change is necessary, performance testing may commence after full regression testing has been completed and the Performance Test Engineer has reviewed the change(s) and made necessary adjustments to test automation.
• All project technical support resources must be identified and available for inquiries and problem resolution.
• In the event that performance testing is suspended due to a discovered defect, resumption criteria will be agreed upon by the Business Analyst and the Performance Test Engineer, and performance testing will not re-commence until the software reaches these criteria.

Minimal Exit Criteria for Performance Testing

• All scheduled performance test activities must be completed.
• All critical defects identified during performance testing must be fixed and re-tested under load.
• If any medium or low-priority defects identified during performance testing are outstanding, then the implementation risk must be signed off as acceptable by the Business Analyst, Quality Analyst, and Business Sponsor.
• All specified performance requirements must be met successfully. If any performance requirements are not met, then the implementation risk must be signed off as acceptable by the Business Analyst, Quality Analyst, and Business Sponsor.
• A full analysis of all performance test results must be completed by the Performance Test Engineer and summarized in a Performance Test Summary Report.

Load Generation Strategies

• Peak: - Gradual ramp up to specified number (n) (e.g. 100) of concurrent users; maintain peak concurrent users for specified time (t) (e.g. one hour); monitor critical system components to observe system response. The objective of this test is to validate that the system components will meet business and technical requirements.
• Breakpoint:- Gradual ramp up to target peak of specified number (n) (e.g. 100) of concurrent users; continued ramp pass peak to a point at which response time exceeds business requirements; monitor critical system components to observe system response. The objective of this test is to stress system components to determine their breakpoint.
• Longevity:- Gradual ramp up to target peak of specified number (n) (e.g. 100) of concurrent users; maintain peak concurrent users for a period of time that would represent at least for specified days (d) (e.g. 5 days) total transactional volume; monitor critical system components to observe system response. The objective of this test is to ensure the system will function at peak load for an extended period.
• Burst:- Immediate start up and login of target peak of specified number (n) (e.g. 100) concurrent users; monitor users for a period of time for specified time (t) (e.g. 20-30 minutes) to ensure that all users are eventually serviced by the system and errors decrease; monitor critical system components to observe system response. The objective of this test is to ensure that the system can handle a spike of volume that may occur following an outage.
• Failover:- Gradual ramp up to a typical production load (e.g. 75% of target peak); maintain peak concurrent users for a period of time (e.g. 2 hours) while critical system components (web servers, application servers, database server) are intentionally failed; monitor critical system components to observe system response. The objective of this test is to ensure the system will perform under various