Software Testing

ETL Testing Process Overview

2009-08-21T15:06:00.001+05:30

This is a compilation of my friend Rekha Ram Burra's experiences while testing ETL applications. I just borrowed this information from him to share publicly on my Blog. Thanks a lot to Rekharam being kind enough to allow me to reuse this content for publishing in my blog.

This is a compilation of the test activities performed while testing an ETL application. Though ETL stands for Extract, Transform and Load, as the final target of the test team is to validate the data in destination database, Test team should only verify data between source and destination. It is not compulsory that Test team should verify how data is extracted and transformed. We can consider this as Black Box Testing.

Assumptions

There are four different databases (on same server or different servers) in place.

At least one Source database system which is on a different server than the below databases
A Delta database that holds source data in tables with schema similar to source table schema
Staging database where the data that is needed for the transformations will reside with any intermediate tables needed
A Factory database where the data after required transformations will be loaded into tables
Warehouse database which is the final database which is a data warehouse

There can be changes to the above architecture. E.g. both the delta and staging databases can be combined to one single database still with different tables for delta and staging. The general architecture would be to maintain Staging, Factory and Warehouse databases separately (if not on different servers) to reduce load on the warehouse.

Test approach

Verify the data type of the columns between source and destination

Between Source and Delta

Test should make sure that the data type of each column of each table is as per the Functional Specification. If no specific details are mentioned in Functional Specification (FS) about the table schema, then test should make sure on the below:

The data type of the source column and the destination column are same
The destination column length is Equal to or Greater than the Source column length

Between Source and Factory

The data type of the source column and the destination column are same
The destination column length is Equal to or Greater than the Source column length
If the data type is different between source and destination then test should make sure that it is documented and no data is lost

Between Factory and Warehouse

The schema of tables will be exactly same between Factory and Warehouse. The below query can be used to check the table schema. Please note, the below are linked queries assuming that the Factory and Warehouse databases are on different servers. You may have to add the servers as linked servers using SP_ADDLINKEDSERVER/ remove the linked servers if the databases are on same server.

DECLARE @FactColCount int, @RepColCount int, @MatchCount int

SELECT @FactColCount = COUNT(*) FROM <FactoryServerName>.<FactoryDatabaseName>.dbo.sysobjects so INNER JOIN <FactoryServerName>.<FactoryDatabaseName>.dbo.syscolumns sc ON so.id = sc.id WHERE so.type = 'U' AND so.name = <Factory Table name to check>

SELECT @RepColCount = COUNT(*) FROM <WarehouseServername>.<WarehouseDatabaseName>.dbo.Sysobjects so INNER JOIN <WarehouseServername>.<WarehouseDatabaseName>.dbo.syscolumns sc ON so.id = sc.id

WHERE so.type = 'U' AND so.name = <Warehouse Table name to check>

SELECT @MatchCount = COUNT(*) FROM <FactoryServerName>.<FactoryDatabaseName>.dbo.sysobjects so1, <FactoryServerName>.<FactoryDatabaseName>.dbo.syscolumns sc1, <WarehouseServername>.<WarehouseDatabaseName>.dbo.sysobjects so2, <WarehouseServername>.<WarehouseDatabaseName>.dbo.syscolumns sc2 WHERE so1.type = 'U' AND so1.id = sc1.id AND so1.name = <Factory Table name to check> AND so2.type = 'U' AND so2.id = sc2.id AND so2.name = <Warehouse Table name to check> AND (sc1.name = sc2.name AND sc1.colorder = sc2.colorder AND sc1.name = sc2.name AND sc1.length = sc2.length AND sc1.isNullable = sc2.isNullable)

IF (@FactColCount <> @RepColCount)

SELECT 'ERROR: The number of columns (' + convert(varchar(20), @FactColCount) + ') in Factory does not match with the number of columns (' + convert(varchar(20), @RepColCount) + ') in Warehouse'

ELSE

IF (@FactColCount <> @MatchCount)

SELECT 'ERROR: The number of columns (' + convert(varchar(20),@RepColCount) + ') in both Factory and Warehouse matches. But some of the attributes does not match'

ELSE

SELECT 'SUCCESS: The tables are identical in structure'

Note: The above query uses SysObjects and SysColumns which are discouraged on SQL 2008. You may consider changing it as per your need

Verifying the data

Check the number of records in source and destination

A simple SELECT COUNT(*) FROM <Table Name> with needed WHERE condition should be enough to check whether there are same number of records on source and destination. But we cannot rely on just this query, reason: assume one record is duplicated into destination instead of another record. In this case, the number of records will be same but there is one record missing from source.

We may encounter two kinds of scenarios:

a. When the data in source and destination tables can be related using a Key

DECLARE @SrcRowCount int, @DestRowCount int

DECLARE @SrcExtraRows int, @DestExtraRows int

SELECT @SrcRowCount = COUNT(<ID Column>) FROM <Source Server>.<Source Database>.dbo.<SourceTable> gcd /*Any Other Join Conditions that are needed WHERE Conditions needed */

SELECT @DestRowCount = COUNT(*) FROM <WarehouseServername>.<WarehouseDatabaseName>.dbo.<WarehouseTableName>

SELECT @SrcExtraRows = COUNT(*) FROM <Source Server>.<Source Database>.dbo.<SourceTable> gcd /* Any Other Join Conditions that are needed WHERE Conditions needed */

AND gcd.<ID Column> NOT IN

(

select <ID Column> from <WarehouseServername>.<WarehouseDatabaseName>.dbo.<WarehouseTableName> /*Do not forget to filter the Zero Key if the table has one*/

)

SELECT @DestExtraRows = COUNT(*) FROM <WarehouseServername>.<WarehouseDatabaseName>.dbo.<WarehouseTableName> WHERE <ID Column> NOT IN

(

SELECT gcd.<ID Column> FROM <Source Server>.<Source Database>.dbo.<SourceTable> gcd /* Any Other Join Conditions that are needed WHERE Conditions needed */

)

IF (@SrcRowCount <> @DestRowCount)

PRINT 'ERROR: The number of rows in destination table does not match with the number of rows in source.'

ELSE IF (@SrcExtraRows > 0)

PRINT 'ERROR: All the records of source table have not migrated to Destination table'

ELSE IF (@DestExtraRows > 0)

PRINT 'ERROR: There are extra Ids in Destination table that are not present in the Source.'

ELSE

PRINT 'SUCCESS: The tables have same row counts and same Ids'

b. When the data in source and destination tables cannot be related using a Key (There may be cases where every time the destination is just truncated and repopulated with the source data). In this case,

Get the count of records from source grouping by all columns
Get the count of records from source grouping by all columns
Compare the count from (i) and ii and it should be same

Verifying Straight Map columns

If there is only one source table for a destination Warehouse table, identify all the columns that are just copied from source and stored in destination. Use the below query to verify the column values

SELECT TOP 10 'Exp StraightMapColumn1' = Source.StraightMapColumn1, 'Act StraightMapColumn1' = Destination.StraightMapColumn1, FROM <Source Server>.<Source Database>.dbo.<Source Table> Source JOIN <WarehouseServername>.<WarehouseDatabaseName>.dbo.<Destination Table> Destination ON Source.<Id Column> = Destination.<Id Column> WHERE Source.StraightMapColumn1 <> Destination.StraightMapColumn1

If there are more than one source table from which few columns are just copied into destination table, we can group the columns by source table and use the above query for each group.

Verifying Derived Columns

Simple derivations like coalesce can be directly done using the same straight map query with a few changes. If there are any complex calculations for a column, then it is always suggested to create a temporary table only with two columns an Id column that can be used to map to the source and destination tables and the actual column that needs to be tested. Following the logic given in Functional Specification (FS) we have to populate the temporary table (This is similar to what dev do but Test also should do it to validate the column values). Once the temporary table is populated, we can use the same above straight map query to compare data between the temporary table and the destination table.

Verifying Zero Key row

There will be a record (may not be in every warehouse) with the surrogate key value as 0. This record contains the default values for each column. This record must be validated with the values given in the Functional Specification (FS).

Conclusion

This document gives a basic idea of the approach that can be followed to test the finally loaded data. There may be a lot other approaches for similar kind of application testing. Once you get used to the approach, you can further refine the approach by automating the test cases to pick up the column names automatically and using Dynamic SQL. Once accustomed with the queries, you can start looking into the code for any kind of discrepancies that might occur due to data (e.g. looking into queries that might generate duplicate data etc…)

Project Vs Product Testing

2009-04-13T16:44:00.001+05:30

Imagine a situation where you are working on testing a project and you have completed testing of the project. You got a new assignment and you are going to test a Product this time.

Now, what kind of shift you need to make in your approach, scope, areas of importance between testing a project and product is what is going to be discussed in this whitepaper.

Project Testing

Normally Project will be specific to single client's requirement. Everything in a project revolves around a single client, his requirements, pain points etc.

A Project can be executed using any of the SDLC models. But the most preferred models for executing a project are

Iterative
Prototype
Agile

If you observe Waterfall model is not the most preferred SDLC for executing a project. This is because, clients requirements will keep on changing as time passes and these changes need to be absorbed by the project team and final delivery should be made as per the changed requirements.

Below listed are the various types of testing to be given importance and should be taken into consideration when testing a Project.

Functionality Testing: Functionality of the application is of utmost importance in case of project. Also the functionality should exactly match with the project specifications or requirements.

User Interface Testing: User interface has to be tested for boundary, tab order, any limitations on data to be entered, punctuations, formats to be followed for data entry, ease of use, Spelling and grammar etc.

Security Testing: Now a day's more emphasis is given on security due to the various security threats a project has to deal with after it is live in production. So the application has to be tested for any existence of security loop holes, un-authorized accesses in both front-end and back-end. Some Examples of security checks are cross site scripting, SQL injection etc.

Performance and Load Testing: These are to be given due importance while testing a project. Performance and benchmarks and expected user Load along with mix of users should be clearly identified and defined before starting the testing, Load and performance test tools are best suited for doing this kind of testing.

Concurrency Testing: Concurrency play vital role in case of financial domain related applications. Even this is important for non-financial domain applications. This testing is little difficult to perform manually. So using automation tools is a good option to do this testing. Concurrency scenarios should be clearly defined and automation tool should be configured properly for getting best results.

Product Testing

When we talk about Product the requirements will be a common pool of requirements gathered from several clients/end users and prioritized and frozen for a release/version.

The most preferred SDLC models for executing a product are

Waterfall
Incremental

Below listed are the various types of testing to be taken into consideration when testing a Product besides normal functional, UI, load and performance testing.

Installation Testing: We should test a product for proper installation and this testing focus on the proper installation of the product based on the instructions provided in the installation guide. The installation should be simple and should involve minimal human intervention for greater customer satisfaction.

Fail Over/Recovery Testing: Every product should handle failures gracefully and recover from the failures with no or minimum loss of data. The main focus in this testing is to simulate the possible failure scenario and observe how the failure is handled. Once the system is back to normal situation how the recovery is made and whether there is any data loss in the failure and recovery process.

Help/User Guide: For any product Help document or the User Guide is heart as the product can be purchased and used by any one. And the first this user will refer to is Help or User guide for using the product or solving any problem while using the product. So these documents should be tested thoroughly for completeness, clarity. All navigations, re-directions, links should be checked for proper functioning in the Help or User Guide.

Portability Testing: In this testing we need to test whether the product works correctly on various configurations, Operating Systems, Hardware etc.

User Friendliness/Usability: In this testing we need to check how easy/difficult for the user to use the product. Can people with disabilities use this product easily? Any tools or options provided for disabled users to use the application easily without any problems?

Copyright Information and Packaging: Good packaging of a product attracts interest of more number of users. And in case of a product Copyright information is of utmost importance. The organization can land into legal problems if copyright information is not proper.

Privacy and Confidentiality: In these days a lot of emphasis given on privacy and Confidentiality of the information handled using a product. So this aspect should be given due importance when testing a product.

Parallel Testing: This type of testing comes into picture and should be given due importance when a product developed in one technology is re-developed or migrated into a new technology. In this testing we will compare the functional output of the new product with output of the old product. We have to compare the functional scenarios and we should get the same output in both versions.

Project Vs Product Testing

Below is a snapshot of some differences between a project and product

*Project*	*Product*
Requirements will be specific to a single client	Requirements will be derived from a common pool of several clients/end users
Iterative, Prototype and Agile SDLC models are most preferred	Waterfall and Incremental SDLC model are most preferred
Requirements will be dynamic and may change very frequently	Requirements will be constant most of the times
Project will be delivered to the client directly	Product will be released in the market and can be purchased by any one
Test Automation is not encouraged in case of a project where the requirements change very frequently	Test Automation will help immensely in case of testing a product
Projects will be planned for the immediate use with less focus on future extendibility or versions	Products will be planned and will have several newer versions and covers several years to come
Project durations will be shorter in comparison with products (Maintenance/ Sustenance are exceptions)	Product durations will be longer generally in comparison with Projects

Conclusion

The conclusions we can derive from the above discussion are

Testing a Product and Project are not one and the same.
Targeted audience for a Product and a Project are different hence the testing strategy and approach should also be different
Different SDLC models are preferred for project and product.
Each testing should focus on different areas of testing.
Automation will help some parts of Project testing and major parts of Product testing.

Basics of Performance Testing

2008-10-31T18:03:00.003+05:30

Concepts and Background

Software testing is the process of executing software and comparing the observed behavior to the desired behavior. The major goal of software testing is to discover errors in the software, with a secondary goal of building confidence in the proper operation of the software when testing does not discover errors.
Testing alone does not produce software quality. Testing ensures that the software functions as it was initially requested by those who dictate the requirements. Therefore, not testing software increases the risk that the software may not comply with the requirements and won't necessarily provide the expected added business value.
In the literature and lexicon of software testing, the terms load testing, performance testing, scalability testing, and stress testing have many (sometimes overlapping) meanings. For the purpose of this document, the basic definition of load testing is: "Testing that involves executing operations simultaneously, at the same rates and with the same other environmental conditions as those that will occur in the field; thus the same interactions and impact of environmental conditions will occur as can be expected in the field."

Goal of Performance Testing

Within the discipline of software testing, the goal of performance testing is to exercise the software and hardware components of a system under varying conditions of load in order to discover errors (bottlenecks, defects, single-points-of-failure, and resource limitations) and to build confidence (responsiveness, robustness, reliability, efficiency, scalability, and resiliency).
Performance testing is based on the practice of modeling the expected usage of a software program or system by simulating multiple users accessing the system's services concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as a web application. However, software systems can be load-tested in other ways also. For example, a system can be forced to transmit or process an extremely large document; or a system can be forced to generate a report based on several years' worth of data.
Performance testing proves that the system-under-test meets (or does not meet) its performance requirements or objectives, specifically those regarding response time, throughput, and utilization under a given (usually, but not necessarily, average or normal) load. Therefore, performance testing activities should seek to measure and evaluate response times, transaction rates, and other load and time sensitive requirements of an application in order to verify that required performance has been achieved.
When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high loads or projected growth levels, it is known as stress testing. The load is usually so great that error conditions are the expected result, although the system should stabilize at levels less than design limits. Stress testing submits the SUT to extremes of workload processing and resource utilization. The general approach is to drive one or more components or the entire system to processing, utilization, or capacity limits by submitting as much workload as is necessary (all at once or increasing incrementally) to achieve that end, even forcing it to its breaking point. The purpose of stress testing is to reveal defects in software that only appear, or are more easily noticed, when the system is stressed to and beyond its limits.
Background testing is the execution of normal functional testing while the SUT is exercised by a realistic work load. This work load is being processed "in the background" as far as the functional testing is concerned. Background testing differs from normal functional testing in that another test driver submits a constant "background" workload to the SUT for the duration of the functional tests. The background workload can be an average or moderate number of simulated users, actively exercising the system. The rationale for doing background testing is to test the system's functionality and responsiveness under more realistic conditions than single-user or simulated single-user functional testing allows.

Minimal Entrance Criteria for Performance Testing

Performance requirements must be finalized and testable, must have passed inspection and must have been signed off by the Business Analyst, Quality Analyst, and Performance Test Engineer.
The Performance Test Approach must be finalized, must have passed inspection and must have been signed off by the Business Analyst, Quality Analyst, and Performance Test Engineer.
The performance testing timeline must be reflective of test development, execution, and analysis/reporting time estimates and resource constraints provided by the Performance Test Engineer.
All developed code must be unit and system tested and free of critical defects. Code development must be completed and signed off as a load-ready release by the development team.
All test hardware and environments must be in place, must have passed integration testing, and must be scheduled for exclusive use during performance tests.
Test hardware, environments, and data-loading must be production-like to ensure accurate performance analysis. If a production-like test environment is not feasible, performance testing may commence if the Business Analyst, Quality Analyst, and the Performance Test Engineer are in full agreement that the risk is manageable.
All code and configuration changes must be frozen. If a code or configuration change is necessary, performance testing may commence after full regression testing has been completed and the Performance Test Engineer has reviewed the change(s) and made necessary adjustments to test automation.
All project technical support resources must be identified and available for inquiries and problem resolution.
In the event that performance testing is suspended due to a discovered defect, resumption criteria will be agreed upon by the Business Analyst and the Performance Test Engineer, and performance testing will not re-commence until the software reaches these criteria.

Minimal Exit Criteria for Performance Testing

All scheduled performance test activities must be completed.
All critical defects identified during performance testing must be fixed and re-tested under load.
If any medium or low-priority defects identified during performance testing are outstanding, then the implementation risk must be signed off as acceptable by the Business Analyst, Quality Analyst, and Business Sponsor.
All specified performance requirements must be met successfully. If any performance requirements are not met, then the implementation risk must be signed off as acceptable by the Business Analyst, Quality Analyst, and Business Sponsor.
A full analysis of all performance test results must be completed by the Performance Test Engineer and summarized in a Performance Test Summary Report.

Load Generation Strategies

Peak: - Gradual ramp up to specified number (n) (e.g. 100) of concurrent users; maintain peak concurrent users for specified time (t) (e.g. one hour); monitor critical system components to observe system response. The objective of this test is to validate that the system components will meet business and technical requirements.
Breakpoint:- Gradual ramp up to target peak of specified number (n) (e.g. 100) of concurrent users; continued ramp pass peak to a point at which response time exceeds business requirements; monitor critical system components to observe system response. The objective of this test is to stress system components to determine their breakpoint.
Longevity:- Gradual ramp up to target peak of specified number (n) (e.g. 100) of concurrent users; maintain peak concurrent users for a period of time that would represent at least for specified days (d) (e.g. 5 days) total transactional volume; monitor critical system components to observe system response. The objective of this test is to ensure the system will function at peak load for an extended period.
Burst:- Immediate start up and login of target peak of specified number (n) (e.g. 100) concurrent users; monitor users for a period of time for specified time (t) (e.g. 20-30 minutes) to ensure that all users are eventually serviced by the system and errors decrease; monitor critical system components to observe system response. The objective of this test is to ensure that the system can handle a spike of volume that may occur following an outage.
Failover:- Gradual ramp up to a typical production load (e.g. 75% of target peak); maintain peak concurrent users for a period of time (e.g. 2 hours) while critical system components (web servers, application servers, database server) are intentionally failed; monitor critical system components to observe system response. The objective of this test is to ensure the system will perform under various

Test Case Design Techniques

2008-08-13T12:38:00.016+05:30

In general a lot of techniques are used while designing/preparing the test cases. These techniques help us to limit the number of inputs to be tested for testing the specified range or boundaries. We will discuss about two of the most prominent of these techniques. These techniques are called

Boundary Value Analysis (BVA)
Equivalence Class Partition

Boundary value analysis/ testing: - A test case design technique for a component in which test cases are designed which include representatives of boundary values. A testing technique using input values at, just below, and just above, the defined limits of an input domain; and with input values causing outputs to be at, just below, and just above, the defined limits of an output domain.

Equivalence Class: - A portion of a component's input or output domains for which the component's behavior is assumed to be the same from the component's specification

Equivalence Partitioning: - A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes

Let's understand these two techniques using a simple example. Below given is a sample specification for a Login page of an application.

Specifications: - 1) Application is having a Login Page containing two text boxes User Name, Password. It also have two command buttons Login, Reset.

2) The Login Text box should accept alphanumeric input only.

3) The password should be a minimum of 6 characters and a maximum of 10 characters. And it should contain at least one special character, One Alphabet and One Number.

Application of Boundary Value Analysis (BVA) technique:-

BVA can be stated in other words as; if L, U are the Lower and Upper limits of a range of input, then the tendency of having a defect is most likely at values L-1, L, L+1, U-1, U, U+1.

Now let us look at the Specification#3. It says that the password should have a length between 6 to 10 characters. In this case

Lower Boundary (L) = 6

Upper Boundary (U) = 10

So we need to consider testing with input having 5 (L-1), 6 (L), 7 (L+1), 9 (U-1), 10 (U), 11 (U+1) characters as these are the value that are most likely to cause a failure/defect.

At a first look, it may not seem to have saved us from creating number of test cases. Now assume that the lower and upper bounds are 2, 100 respectively. In order to design test cases for this kind of input range imagine the test cases you need to design without using the BVA technique. At a minimum you need to test with 100 inputs. If you apply BVA for this scenario the inputs reduce to 1, 2, 3, 99, 100, and 101. May be we can check for one more input between 3, 99 to be on safe side. So the total inputs we need to use reduced to 7. Do you see the effect of applying BVA now?

Application Equivalence Class Partition technique:-

In simple terms Equivalence Class Partition can be stated as testing with one representative from each equivalence class is enough to prove the success or failure of testing with the entire equivalence class.

Let us look at the Specification#2. It says that Login text box should accept only alphanumeric input. This means no special characters allowed. Once special character comes into picture the input becomes Invalid. As long as you use combination of alphabets and Numerals (Alphanumeric) the input is Valid.

Now, we can divide our input into two partitions Valid and Invalid Input. This forms our first level of partitions.

Let's look at the valid inputs now. We can use only Numbers, Only Alphabets, and combination of both Numbers and Alphabets as input. This leads us to the new partitions Numbers Only Input, Alphabets Only Input, Alphanumeric Input. Like this we can go on identifying different partitions based on the specification and need.

So if we put all of this at one place we get the following partitions.

VALID INPUT
Numbers Only
Alphabets Only
Alphanumeric
INVALID INPUT
Special Characters Only
Alphanumeric + Special Characters

Now, if you test one input in the Numbers Only partition, we can safely apply the result of this test to all values in the same partition. So testing with one input from each partition is enough to cover testing of the entire partition. In our example we can test with 5 inputs in total and identify the outcome of all these five partitions. So, this way we can reduce our effort to create number of test cases.

Now two more definitions that are of use and indirectly related to the concepts we discussed above.

Coverage Testing (Test Coverage):- Coverage testing is concerned with the degree to which test cases exercise or cover the logic (source code) of the software module or unit. It is also a measure of coverage of code lines, code branches and code branch combinations

Code Coverage: - An analysis method that determines, which parts of the software code have been executed (covered) by the test case suite and which parts of code have not been executed and therefore may require additional attention.

Project Estimation Techniques

2008-06-04T18:35:00.020+05:30

Introduction:

Estimation is an essential part of the project management methodology. Estimation is used of a number of purposes:

To justify the project, particularly at the proposal stage.
To compare costs with the anticipated benefits.
To secure the resources required to successfully deliver the project.
To enforce the disciplines needed to make the project succeed.
To ensure that the support impact of the project is fully understood.
To inform and improve the software development process used.

Overview of Estimation process:

The best people to undertake estimation are the staff who is going to do the work. The staff chosen to produce an estimate is drawn from MIS, customer areas and/or service partners who have relevant experience of similar previous projects or tasks in the business area. Estimation technique is based on three components:

a) Expert judgment: i.e. consultation with qualified experts from within MIS and the organizations business and service partners. This will be supplemented by expert input from software suppliers and consultants on a need basis.

b) Experience: i.e. comparison of the proposed project or task with one or more previously completed developments.

c) Task Decomposition: i.e. decomposing the project into components, i.e. the Work Breakdown Structure, and estimating these components individually to produce an overall estimate.

The estimates are validated through peer review and are backed up by an experienced Project Manager who takes overall responsibility for the total. Estimated are reviewed throughout the project lifecycle. Estimates, and the process followed to produce them, are transparent and consistent across all MIS projects.

Base and Contingency method

In this technique all estimates will have two components the base and the contingency. The base is the minimum expected time required, i.e. when everything goes well. The contingency is the amount of trust placed on the base when risks are taken into account. The contingency is generally expressed as % of the base estimate.

For a straight forward task contingency may be between 10 to 20% and for a high risk task it may go to a level of 50% or more. Separating the base and contingency will make the estimations clear and easier. We can derive a base figure based on the experience without talking all the issues or risks that may occur. And to determine the level or % of contingency we can use any risk analysis techniques.

The contingency is based on the possibility of a risk occurring (how likely the risk going to occur) and weightage given to that risk (The effect/impact of the risk). Project contingency can be estimated in terms of money, resources or a mixture of both. Resource contingency is managed by adding a factor to individual task estimates. The Project Manager will also consider project wide risks and undertake a risk analysis to determine the correct amount of contingency to allow for them.

For providing estimates using this technique the following steps to be followed.

Decompose the project into a list of tasks i.e. Work Breakdown Structure.
Estimate the base and contingency (task level) for each task.
Add all the estimations together.
Add the project level contingency.

An important point to note here is we should not do double-counting i.e. adding contingency to both task and project level estimated for the same risk. Below is a sample to show estimation using this technique.

----------------------------------------------------
Task Name         Base     Contingency     Total
----------------------------------------------------
Build Login UI    8 Hrs    0.8 Hrs         8.8 Hrs
                           (10% of base)

Login Validation 16 Hrs    3.2 Hrs        19.2 Hrs
SP                         (20% of base)
-----------------------------------------------------
Total            24 Hrs    4.0 Hrs        28.0 Hrs
-----------------------------------------------------

Total Task level estimate (Including Task Level Contingency) = 28.0 Hours Contingency at Project Level = 6.0 Hours (Based on the risk analysis) Overall Estimate = 34.0 Hours (Including both task level and project level contingency)

Three Point Estimation method

Another estimation technique that can be used in place of base and contingency method is three point estimation technique. This technique is based statistical methods, and in particular normal distribution.

For every estimate we produce three numbers/values: a = the best case estimate m = the most likely estimate b = the worst case estimate These values are used to calculate and E value for the estimate and a Standard Deviation using the following formulae: E = a + (4 * m) + (b / 6) SD = (b – a) / 6 E is the sensible estimate for the talk and is a weighted average which taken into account both the most optimistic and pessimistic estimated provided. SD is the variability or uncertainty in the task estimate, and in essence is a measure of risk.

We can use E and SD to convert these estimates to Confidence Levels as follows.

Confidence Level in E value is approximately 50%
Confidence Level in E value + SD is approximately 70%
Confidence Level in E value + 2 * SD is approximately 95%
Confidence Level in E value + 3 * SD is approximately 99.5%

For providing estimates using this technique the following steps to be followed.

Decompose the project into a list of tasks i.e. Work Breakdown Structure.
Estimate the E value and SD for each task.
Calculate the E value for the total project work as E (Project work) = ∑ E (Task)
Calculate the SD value for the project work as SD (Work) = √ ∑ SD (Task)2
Calculate the E value for the project duration as E (Duration) = ∑ E (Critical Path Task)
Calculate the E value for the project duration as SD (Duration = √ ∑ SD (Critical Path Task)2
Depending on the confidence level required on estimations use the E Value and multiple of SD to calculate the final estimate.

Concurrency Testing

2008-05-22T16:29:00.014+05:30

Concurrency testing can be defined as the behavior and state of data when different users access the same data. The transition and business processing for the data is also an important aspect in this testing. Concurrency testing can be done by Team effort and only team effort. Users should define a strategy how many users will access and what set of data. Once this is freezed, the hardware requirements to do the same (preferable the hardware used should be of the same specifications). In any applications Concurrency testing is one of the important phases. Most of the times due to time constraints this phase is ignored or this is not done at all. Check the below example which will display the importance of doing concurrency testing. Mr. A is an account holder in a bank B. He issued a check to one of his friends Mr. C for 50,000/- on a certain date. Mr. A went to deposit some money in bank B on the same day when he issued a check. At this time the balance of Mr. A in bank B was 10,000/-. Now, unfortunately for him two tellers accessed Mr.A’s data at the same time. At 10.15 AM Teller A --> Mr. A’s Balance: 60,000/- Teller B --> Mr. A’s Balance: 60,000/- At 10.17 AM Teller A --> Deposit: 40,000/- --> Mr. A’s Balance: 100,000/- --> Updates At 10.20 AM Teller B --> Withdraws: 50,000/- --> Mr. A’s Balance: 10,000/- --> Updates Finally the balance of Mr. A is 10,000/-!!! The possibility of this situation occurring may be 1 out of 1000 chances. But, that 1 instance is also pretty important and may cause high customer dissatisfaction. As a result of this the customer may lose the confidence on the organization he is operating with (Bank B in the above example).

The problem caused here is due to the improper use of Database locks at the table level and most importantly at row level. Because of these kind of a problems concurrency testing is given more importance and should be performed on all systems that have scope for concurrent accessing of same data by multiple users.

We need to deal with certain issues before we go for concurrency testing, some of the important issues are

Simulation of real time scenario: It is very difficult to simulate the real time scenarios in test labs or test environment.

Accuracy in actions: Getting data and saving at the specific time is a tricky thing to do practically.

Verification of Data: The data need to be verified in each phase like at Start of test, before and after the actual transaction/change takes place etc.

Load on the Application Server: It is difficult to determine the exact load to be simulated on the application server in order to carry out the concurrency testing and to make the transaction/change that will cause the failure of the system.

Scalability of the Application (Only in high end applications): The application should be scalable enough to sustain the load that is created during the concurrency testing.

Data persistency: Data persistency just means that the data is stored somewhere, usually DB or file, for whatever time necessary. It is important to know where and how the data is stored as part of performing concurrency testing.

Below checklist can be of use while performing the concurrency testing

Define the data set to be tested for concurrency (including screens also)
Outline a strategy who and how
Define Hardware
Execute test for Phase 1 (Data changed and updated)
Execute test for Phase 2 (Same data changed and updated)
Check Data at the end of each phase

Note: Concurrency testing is often confused with Load/Stress testing. The ability to with stand a huge amount of data or cater to huge number of users corresponds to Stress testing.

Software Test Automation

2008-05-15T17:52:00.011+05:30

Software Test automation is becoming prominent day by day. This article is written to give some details about automation based on my experiences.

The process of testing an application using a tool which will simulate the actions performed by a normal tester can be called as Test Automation. The tester will use techniques like record/playback provided by most of the test automation tools to generate some base code. Then he will modify/customize the code generated by tool based on his need. In simple terms Code that tests code is what automation all about!

Why Automation is becoming more and more important now a days? Every day new technologies are coming into market which are easy to use and have very powerful features. As a result of this the time taken for developing software applications is reducing drastically. In order to keep up with the speed of development, the software testing industry is looking towards automation as the solution for this problem.

Does automation really save testing time? The answer to this question is Yes and No. If the automated scripts are used only once or twice, then there is no way you are going to save testing time. Instead you will be spending more time to create the test scripts, maintain them etc. Then, how it is going to save time? When you execute the automated scripts for number of times then you are going to save the testing time. See the below example Example: Time required to test application A manually = 50 Hours Time required to automate the testing for Application A = 100 Hours Time required to test application A using automated scripts = 5 Hours

Case#1: Automated scripts executed for 2 times Total effort spent is = 110 Hours (100 + 2 x 5=110) Time required to test the application A manually 2 times = 100 Hours (50 x 2=100)

In this case you spent extra 10 Hours (100 – 110) for testing through automation over manual testing. Now see case#2

Case#2: Automated scripts executed for 3 times Total effort spent is = 115 Hours (100 + 3 x 5=115) Time required to test the application A manually 3 times = 150 Hours (50 x 3=150)

In this case you saved 35 Hours (150 – 115) through automation over manual testing.

The hours saved will increase as the number of time you use automated scripts increases.

What are the other advantages by using test automation? Automation will save money. In the example we have seen how time can be saved using automation number of times. When you save time, in turn you are saving money, right?

Your tests will be consistent using automation. It is common for a human being to overlook or miss something when the same work is done number of times. With automation you can avoid this. Your tests with run with the same consistency how many number of times you run them.

Testing becomes independent of the tester. By using automation we can delink the testing from the dependency of test execution from a test engineer. Once the automated scripts are ready any test engineer can run them with minimal training or knowledge about the test scripts. If the automation code is written carefully, tests can be run even in the absence of test engineer.

Load and Performance testing is difficult to carry out manually. In case you want to test an application with 100 users accessing it simultaneously and doing various actions. Imagine how difficult it is to get 100 systems, install the application on them, and get 100 people to work on these systems at the same time. But, if you are using the performance or load testing tools, you can do this kind of test comfortably with only 1 or 2 resources.

What is Break-even Point, ROI (Return on Investment) in test automation? In economics, specifically cost accounting, the break-even point (BEP) is the point at which cost or expenses and revenue are equal: there is no net loss or gain, and one has "broken even". The same applies in case of Test Automation. We will say we achieved BEP when the cost for doing automation equals the saving achieved by automated scripts execution. While calculating the cost for automation lot of things will be considered like license cost of the automation tool, hours spent for developing automated scripts, time spent for debugging/customizing/maintaining automation scripts, time taken for executing the automated scripts etc.,

ROI stands for Return on Investment. The following simple formula can be used for calculating the ROI % % ROI = (benefits / costs) x 100 Once you cross the BEP, then you will start getting ROI. This is the very reason why automation requires some initial investment, and it will be beneficial over a period of time.

Testing Zero documented Applications

2008-05-06T11:34:00.002+05:30

Abstract

Being in testing domain, each one of us may come across a situation to test an application; which does not have any documentation.

What can we do in such situations? Can we say we cannot test the application, without having any documentation? Can we…? So, how to design test strategy for such an application?

This white paper tried to compile some methods that can be applied in order to design test strategy for applications which do not have any documentation.

Introduction

I was sometimes asked to test an application which has zero documentation. I guess the people working in testing domain will come across this situation time and again. In this document I tried to compile some means and ways that can be used /applied in such situations and will help in designing the testing strategy.

Background Being in testing domain, each one of us may come across a situation to test an application; which does not have any documentation.

What can we do in such situations? Can we say we cannot test the application, without having any documentation? We cannot do that…right? So, how can we test such an application?

Ask yourself the following questions

Whether the application under test is a project or a product?
Is there any other application available anywhere with same or similar functionality?
Who has developed this application? Are the developers reachable?
Who are the targeted users of this application? Can they be contacted?

Carefully note answers to the above questions as you find them.

Approach#1: Product with Zero documentation Step#1: Assume that the application you are asked to test is a product. In this case we can rule out the option of contacting the targeted users.

Step#2: So, try the other option we have, who developed this application? After some research (varies from couple of hours to a day) you identified that the people who developed this application cannot be reached.

Step#3: You feel like you are dead locked? Don't get frightened. We still have one more option; is there any other application available anywhere with same or similar functionality?

Step#4: Good god someone somewhere in the world created a similar product is what you found after some research on web. Bingo, you got a way out.

Step#5: Try to get any documentation that product has to offer or refer the help of that product to understand what that product does. Now, based on the knowledge you gained on the other product design testing of your parent application.

Example: You are asked to test a document editing software, whose functionality is similar to Microsoft Word. In this example you can refer to the Microsoft Word Help and get a feel of what is does and what it is used for.

Step#6: Based on the information you gathered, you can design testing of your document editing software.

Note: In case you found out that you are asked to test a product and you are not able to find any other product with same or similar functionality. What should we do now? Check the section Approach#3: Shelved Project with Zero documentation and try the approach mentioned there to design your testing strategy.

Approach#2: Old Project with Zero documentation

Step#1: Assume that the application you are asked to test is a project developed long time back and is currently used by a group of users. In this case we have a very thin chance of contacting the developers.

Step#2: Also, there is a very less chance of getting the information is there any other application available anywhere with same or similar functionality?

Step#3: Hmm…what are we supposed to do now? You identified that there are some users who are using this application. There you are!!!

Step#4: Contact the users who are using this application. Ask them to explain you what they do with this application. You can have a demonstration session with them. Ask them questions about the application. Make a note of the information they provide. Now, design testing of your application based on this information.

Approach#3: Shelved Project with Zero documentation

Step#1: Assume that the application you are asked to test is a project developed long time back and is not currently used by any one. It is kind of a shelved application.

Step#2: In this case we can rule out the options of contacting users, developer. Also we have minute chances of getting information about a similar project anywhere else.

Step#3: Gosh…all roads are blocked. What is my next step?

Step#4: Well, in this case we have to rely completely on our exploration and learning capabilities? Do you have the shades of a detective? Are you a quick learner?

Step#5: Install the application or catch hold of an environment which contains your application and you are allowed to play around with the application as well as the environment.

Step#6: Start hitting each and every button you see on the application, observe what is happening. Make a note of what you understand from that. I am tired of clicking the buttons…

Step#7: Now, for a change try clicking on all the menus and sub menus that are visible on the application and note down what is happening. In this way try to play around with each object that is available in the application and make a note of what you have observed.

Step#8: Ah!!! Now, I know what will happen when I do anything with an object on the application! Wait, I see some link between these items!! Seems like I can derive business scenarios or workflows from what I did in the application!!!

Step#9: Well done, design your testing as per the business scenarios, workflows you were able to extract from your exploration.

Note: The above approach can be tried for applicability for any other situation or scenario not covered in this document

Conclusion

Once you get an idea of the various objects existing in the application under test you can use all the testing concepts like Boundary Value Analysis (BVA), Equivalence Partitioning, negative and positive testing etc. to design the test cases. Also you can determine the scope of testing, types of testing to be done like functional, User Interface etc.

Now, you gathered handful of information by applying one of the techniques mentioned above. Beware not to repeat the same mistake what others did. The reason for me to write this whitepaper, Zero documentation!!!

Document whatever information you gathered during your research or exploration. Make sure to include the following items.

Purpose of the application
Targeted users of the application and what they do with it.
Business scenarios/workflows you were able to identify.
Complex areas of application, the areas to emphasize during testing.
Testing strategy you followed including types of testing you have done.
Any other information of importance you identified.
Any special notes or instructions you want to mention.

Do not forget to add a Note saying that the information mentioned in this document is purely based on your exploration and understanding.

If possible get this document reviewed by someone. If the review is not possible add another note to say that this is un-reviewed information in order to avoid any problems in future.

Hurray, you understood the application, able to design test strategy, produced some documentation. Kudos to you!!!

Software Testing Glossary

2008-05-05T15:46:00.001+05:30

Acceptance criteria The expected results or performance characteristics that define whether the test case passed or failed. Acceptance Testing / User Acceptance Testing An acceptance test is a test that a user/sponsor and manufacturer/producer jointly perform on a finished, engineered product/system through black-box testing (i.e., the user or tester need not know anything about the internal workings of the system). It is often referred to as a(n) functional test, beta test, QA test, application test, confidence test, final test, or end user test Accessibility Testing Verifying a product is accessible to the people having disabilities (deaf, blind, mentally disabled etc.). Ad-hoc Testing Testing carried out using no recognized test case design technique with some knowledge of the application. Agile Testing Testing practice for projects using agile methodologies, treating development as the customer of testing and emphasizing a test-first design paradigm Alpha Testing In software development, testing is usually required before release to the general public. This phase of development is known as the alpha phase. Testing during this phase is known as alpha testing. In the first phase of alpha testing, developers test the software using white box techniques. Additional inspection is then performed using black box or grey box techniques. Arc Testing / Branch Testing A test case design technique for a component in which test cases are designed to execute branch outcomes. A test method satisfying coverage criteria that require that for each decision point, each possible branch is executed at least once. AUT Application Under Test. The application which we are going to test is called AUT. Authorization Testing Involves testing the systems responsible for the initiation and maintenance of user sessions. This will require testing the Input validation of login fields, Cookie security, and Lockout testing .This is performed to discover whether the login system can be forced into permitting unauthorized access. The testing will also reveal whether the system is susceptible to denial of service attacks using the same techniques. Back-to-back testing Testing in which two or more variants of a component or system are executed with the same inputs, the outputs compared, and analyzed in cases of discrepancies Basis Path Testing A white box test case design technique that uses the algorithmic flow of the program to design tests Benchmark Testing Tests that use representative sets of programs and data designed to evaluate the performance of computer hardware and software in a given configuration Beta Testing / Field Testing Once the alpha phase is complete, development enters the beta phase. Versions of the software, known as beta versions, are released to a limited audience outside of the company to ensure that the product has few faults or bugs. Beta testing, is generally constrained to black box techniques although a core of test engineers are likely to continue with white box testing in parallel to the beta tests. Big Bang Testing Integration testing where no incremental testing takes place prior to all the system's components being combined to form the system. Black Box Testing / Functional Testing Black box testing, concrete box or functional testing is used to check that the outputs of a program, given certain inputs, conform to the functional specification of the program. It performs testing based on previously understood requirements (or understood functionality), without knowledge of how the code executes. Bottom-up Testing An approach to integration testing where the lowest level components are tested first; then used to facilitate the testing of higher level components. The process is repeated until the component at the top of the hierarchy is tested. Boundary value analysis/ testing A test case design technique for a component in which test cases are designed which include representatives of boundary values. A testing technique using input values at, just below, and just above, the defined limits of an input domain; and with input values causing outputs to be at, just below, and just above, the defined limits of an output domain. Breadth Testing A test suite that exercises the full functionality of a product but does not test features in detail Bug Bugs arise from mistakes and errors, made by people, in either a program's source code or its design that prevents it from working correctly or produces an incorrect result Business process-based testing An approach to testing in which test cases are designed based on descriptions and/or knowledge of business processes CAST Computer Aided Software Testing Code Coverage An analysis method that determines which parts of the software code have been executed (covered) by the test case suite and which parts of code have not been executed and therefore may require additional attention. Compatibility Testing Testing whether the system is compatible with other systems with which it should communicate. Component Testing Testing of individual software components. Concurrency Testing Multi-user testing geared towards determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores Conformance Testing / Compliance Testing / Standards Testing Conformance testing or type testing is testing to determine whether a system meets some specified standard. To aid in this, many test procedures and test setups have been developed, either by the standard's maintainers or external organizations, specifically for testing conformance to standards. Conformance testing is often performed by external organizations; sometimes the standards body themselves, to give greater guarantees of compliance. Products tested in such a manner are then advertised as being certified by that external organization as complying with the standard Context Driven Testing The context-driven school of software testing is flavor of Agile Testing that advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now. Conversion Testing / Migration Testing Testing of programs or procedures used to convert data from existing systems for use in replacement systems. Coverage Testing Coverage testing is concerned with the degree to which test cases exercise or cover the logic (source code) of the software module or unit. It is also a measure of coverage of code lines, code branches and code branch combinations Cyclomatic Complexity A measure of the logical complexity of an algorithm, used in white-box testing Data flow Testing Testing in which test cases are designed based on variable usage within the code. Data integrity and Database integrity Testing Data integrity and database integrity test techniques verify that data is being stored by the system in a manner where the data is not compromised by updating, restoration, or retrieval processing Data-Driven Testing Testing in which the action of a test case is parameterized by externally defined data values, maintained as a file or spreadsheet. A common technique in Automated Testing Decision condition testing A white box test design technique in which test cases are designed to execute condition outcomes and decision outcomes Decision table testing A black box test design techniques in which test cases are designed to execute the combinations of inputs and/or stimuli (causes) shown in a decision table Decision testing A white box test design technique in which test cases are designed to execute decision outcomes Defect An anomaly, or flaw, in a delivered work product. Examples include such things as omissions and imperfections found during early lifecycle phases and symptoms of faults contained in software sufficiently mature for test or operation. A defect can be any kind of issue you want tracked and resolved. Defect density The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, e.g. lines-of code, number of classes or function points) Dependency Testing Examines an application's requirements for pre-existing software, initial states and configuration in order to maintain proper functionality Depth Testing A test that exercises a feature of a product in full detail. Design based Testing Designing tests based on objectives derived from the architectural or detail design of the software (e.g., tests that execute specific invocation paths or probe the worst case behavior of algorithms). Development testing Formal or informal testing conducted during the implementation of a component or system, usually in the development environment by developers Documentation Testing Testing concerned with the accuracy of documentation. Dynamic Testing Testing software through executing it. Efficiency testing The process of testing to determine the efficiency of a software product End-to-end Testing Test activity aimed at proving the correct implementation of a required function at a level where the entire hardware/software chain involved in the execution of the function is available. Endurance Testing Checks for memory leaks or other problems that may occur with prolonged execution Equivalence Class A portion of a component's input or output domains for which the component's behavior is assumed to be the same from the component's specification Equivalence partition Testing A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes. Equivalence Partitioning A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes Exhaustive Testing Testing which covers all combinations of input values and preconditions for an element of the software under test Exploratory Testing This technique for testing computer software does not require significant advanced planning and is tolerant of limited documentation for the target-of-test. Instead, the technique relies mainly on the skill and knowledge of the tester to guide the testing, and uses an active feedback loop to guide and calibrate the effort. This can be considered as ad-hoc testing done with out any application knowledge. The tested will learn about the application while testing it. Failure The inability of a system or component to perform its required functions within specified performance requirements. A failure is characterized by the observable symptoms of one or more defects that have a root cause in one or more faults. Fault An accidental condition that causes the failure of a component in the implementation model to perform its required behavior. A fault is the root cause of one or more defects identified by observing one or more failures. Fuzz Testing Fuzz testing is a software testing technique. The basic idea is to attach the inputs of a program to a source of random data. If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior. Gamma Testing Gamma testing is a little-known informal phrase that refers derisively to the release of "buggy" (defect-ridden) products. It is not a term of art among testers, but rather an example of referential humor. Cynics have referred to all software releases as "gamma testing" since defects are found in almost all commercial, commodity and publicly available software eventually. Gorilla Testing Testing one particular module, functionality heavily Grey Box Testing The typical grey box tester is permitted to set up or manipulate the testing environment, like seeding a database, and can view the state of the product after their actions, like performing a SQL query on the database to be certain of the values of columns. It is used almost exclusively of client-server testers or others who use a database as a repository of information, or who has to manipulate XML files (DTD or an actual XML file) or configuration files directly, or who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. GUI Testing GUI testing is the process of testing a graphical user interface to ensure it meets its written specifications Heuristic evaluations Heuristic evaluations are one of the most informal methods of usability inspection in the field of human-computer interaction. It helps identifying the usability problems in a user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the "heuristics"). High Order Tests Black-box tests conducted once the software has been integrated Incremental Testing "Integration testing where system components are integrated into the system one at a time until the entire system is integrated. Installation Testing Installation testing can simply be defined as any testing that occurs outside of the development environment. Such testing will frequently occur on the computer system the software product will eventually be installed on. While the ideal installation might simply appear to be to run a setup program, the generation of that setup program itself and its efficacy in a variety of machine and operating system environments can require extensive testing before it can be used with confidence Integration Testing Integration testing is the phase of software testing in which individual software modules are combined and tested as a group. It follows unit testing and precedes system testing. Interface Testing Testing conducted to evaluate whether systems or components pass data and control correctly to each other. Interoperability testing The process of testing to determine the interoperability of a software product Invalid testing Testing using input values that should be rejected by the component or system Isolation Testing Component testing of individual components in isolation from surrounding components, with surrounding components being simulated by stubs Keyword driven Testing A scripting technique that uses data files to contain not only test data and expected results, but also keywords related to the application being tested. The keywords are interpreted by special supporting scripts that are called by the control script for the test Load Testing Load testing is the act of testing a system under load. It generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently. This testing is most relevant for multi-user systems, often one built using a client/server model, such as web servers Localization Testing This term refers to making software specifically designed for a specific locality Logic coverage Testing / Logic driven Testing / Structural test case design Test case selection that is based on an analysis of the internal structure of the component. Also known as white box testing Loop Testing A white box testing technique that exercises program loops Maintainability Testing / Serviceability Testing Testing whether the system meets its specified objectives for maintainability. Maintenance testing Testing the changes to an operational system or the impact of a changed environment to an operational system Model Based Testing Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test. Monkey Testing A monkey test is a unit test that runs with no specific test in mind. The monkey in this case is the producer of any input. For example, a monkey test can enter random strings into text boxes to ensure handling of all possible user input or provide garbage files to check for loading routines that have blind faith in their data. The name 'monkey' comes from the adage that ‘a thousand monkeys at a thousand typewriters will eventually type out the entire works of Shakespeare’ Mutation testing A testing methodology in which two or more program mutations are executed using the same test cases to evaluate the ability of the test cases to detect differences in the mutations N+ Testing A variation of Regression Testing. Testing conducted with multiple cycles in which errors found in test cycle N are resolved and the solution is retested in test cycle N+. The cycles are typically repeated until the solution reaches a steady state and there are no errors Negative Testing / Dirty Testing Testing aimed at showing software does not work. Operational Testing Testing conducted to evaluate a system or component in its operational environment. Pair testing Two testers work together to find defects. Typically, they share one computer and trade control of it while testing Parallel Testing The process of feeding test data into two systems, the modified system and an alternative system (possibly the original system) and comparing results. Path coverage Metric applied to all path-testing strategies: in a hierarchy by path length, where length is measured by the number of graph links traversed by the path or path segment; e.g. coverage with respect to path segments two links long, three links long, etc. Unqualified, this term usually means coverage with respect to the set of entry/exit paths. Often used erroneously as synonym for statement coverage Path Testing Testing in which all paths in the program source code are tested at least once. Penetration Testing The portion of security testing in which the evaluators attempt to circumvent the security features of a system Performance Testing Performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. Performance testing can serve different purposes. It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly Portability Testing Testing aimed at demonstrating the software can be ported to specified hardware or software platforms. Post-conditions Cleanup steps after the test case is run, to bring it back to a known state. Precondition Dependencies those are required for the test case to run Progressive Testing Testing of new features after regression testing of previous features Quality Control Quality control and quality engineering are involved in developing systems to ensure products or services are designed and produced to meet or exceed customer requirements and expectations Ramp Testing Continuously raising an input signal until the system breaks down Random Testing Testing a program or part of a program using test data that has been chosen at random Recovery Testing Confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions Regression Testing Regression testing is any type of software testing which seeks to uncover bugs that occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Release Candidate A pre-release version, which contains the desired functionality of the final version, but which needs to be tested for bugs Reliability Testing Testing to determine whether the system/software meets the specified reliability requirements. Requirements based Testing Designing tests based on objectives derived from requirements for the software component Resource utilization testing The process of testing to determine the Resource-utilization of a software product Risk-based testing Testing oriented towards exploring and providing information about product risks Sanity Testing Brief test of major functional elements of a piece of software to determine if its basically operational Scalability Testing Performance testing focused on ensuring the application under test gracefully handles increases in work load Scenario Testing A scenario test is a test based on a hypothetical story used to help a person think through a complex problem or system. They can be as simple as a diagram for a testing environment or they could be a description written in prose. Security Testing Tests focused on ensuring the target-of-test data (or systems) are accessible only to those actors for which they are intended. Session-based Testing Session-based testing is ideal when formal requirements are non present, incomplete, or changing rapidly. It can be used to introduce measurement and control to an immature test process, and can form a foundation for significant improvements in productivity and error detection. It is more closely related to exploratory testing. It is a controlled and improved ad-hoc testing that is able to use the knowledge gained as a basis for ongoing, product sustained improvement Simulator A device, computer program or system used during testing, which behaves or operates like a given system when provided with a set of controlled inputs Smart testing Tests that based on theory or experience are expected to have a high probability of detecting specified classes of bugs; tests aimed at specific bug types Smoke Testing A sub-set of the black box test is the smoke test. A smoke test is a cursory examination of all of the basic components of a software system to ensure that they work. Typically, smoke testing is conducted immediately after a software build is made. The term comes from electrical engineering, where in order to test electronic equipment, power is applied and the tester ensures that the product does not spark or smoke. Soak Testing Running a system at high load for a prolonged period of time. For example, running several times more transactions in an entire day (or night) than would be expected in a busy day, to identify and performance problems that appear after a large number of transactions have been executed Soap-opera testing A technique for defining test scenarios by reasoning about dramatic and exaggerated usage scenarios. When defined in collaboration with experienced users, soap operas help to test many functional aspects of a system quickly and-because they are not related directly to either the systems formal specifications, or to the systems features-they have a high rate of success in revealing important yet often unanticipated problems. Software Quality Assurance Software testing is a process used to identify the correctness, completeness and quality of developed computer software. Actually, testing can never establish the correctness of computer software, as this can only be done by formal verification (and only when there is no mistake in the formal verification process). It can only find defects, not prove that there are none. Stability Testing Stability testing is an attempt to determine if an application will crash. State Transition Testing A test case design technique in which test cases are designed to execute state transitions. Statement Testing Testing designed to execute each statement of a computer program. Static Testing Analysis of a program carried out without executing the program Statistical Testing A test case design technique in which a model is used of the statistical distribution of the input to construct representative test cases. Storage Testing Testing whether the system meets its specified storage objectives. Stress Testing Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Stress testing a subset of load testing. Structural Testing White box testing, glass box testing or structural testing is used to check that the outputs of a program, given certain inputs, conform to the structural specification of the program SUT System Under Test. Syntax Testing A test case design technique for a component or system in which test case design is based upon the syntax of the input. System Testing System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of Black box testing Technical Requirements Testing Testing of those requirements that do not relate to functionality i.e. performance, usability, etc. Test Approach The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) project's goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed Test Automation Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Test Bed An execution environment configured for testing. May consist of specific hardware, OS, network topology, configuration of the product under test, other application or system software, etc. Same as Test environment Test Case The specification (usually formal) of a set of test inputs, execution conditions, and expected results, identified for the purpose of making an evaluation of some particular aspect of a Target Test Item. Test Cycle A formal test cycle consists of all tests performed. In software development, it can consist of, for example, the following tests: unit/component testing, integration testing, system testing, user acceptance testing and the code inspection. Test Data The definition (usually formal) of a collection of test input values that are consumed during the execution of a test, and expected results referenced for comparative purposes Test Driven Development Test-driven development (TDD) is a Computer programming technique that involves writing tests first and then implementing the code to make them pass. The goal of test-driven development is to achieve rapid feedback and implements the "illustrate the main line" approach to constructing a program. This technique is heavily emphasized in Extreme Programming. Test Driver A program or test tool used to execute a test. Also known as a Test Harness Test Environment The hardware and software environment in which tests will be run, and any other software with which the software under test interacts when under test including stubs and test drivers. Test Harness In software testing, a test harness is a collection of software tools and test data configured to test a program unit by running it under varying conditions and monitor its behavior and outputs. Test Idea A brief statement identifying a test that is potentially useful to conduct. The test idea typically represents an aspect of a given test: an input, an execution condition or an expected result, but often only addresses a single aspect of a test. Test Log A collection of raw output captured during a unique execution of one or more tests, usually representing the output resulting from the execution of a Test Suite for a single test cycle run. Test Plan A document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. Test Procedure The procedural aspect of a given test, usually a set of detailed instructions for the setup and step-by-step execution of one or more given test cases. The test procedure is captured in both test scenarios and test scripts Test Report A document that summarizes the outcome of testing in terms of items tested, summary of results, effectiveness of testing and lessons learned. Test Scenario A sequence of actions (execution conditions) that identifies behaviors of interest in the context of test execution. Test Script A collection of step-by-step instructions that realize a test, enabling its execution. Test scripts may take the form of either documented textual instructions that are executed manually or computer readable instructions that enable automated test execution. Test Specification A document specifying the test approach for a software feature or combination or features and the inputs, predicted results and execution conditions for the associated tests Test Strategy Defines the strategic plan for how the test effort will be conducted against one or more aspects of the target system. Test Suite A package-like artifact used to group collections of test scripts, both to sequence the execution of the tests and to provide a useful and related set of Test Log information from which Test Results can be determined Test Tools Computer programs used in the testing of a system, a component of the system, or its documentation Testability The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met Thread Testing A variation of top-down testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by successively lower levels Top-down testing An incremental approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested Traceability Matrix A document showing the relationship between Test Requirements and Test Cases and defects Unit Testing A unit test is a procedure used to verify that a particular module of source code is working properly Usability Testing Usability testing is a means for measuring how well people can use some human-made object (such as a web page, a computer interface, a document, or a device) for its intended purpose, i.e. usability testing measures the usability of the object. If usability testing uncovers difficulties, such as people having difficulty understanding instructions, manipulating parts, or interpreting feedback, then developers should improve the design and test it again Use case testing A black box test design technique in which test cases are designed to execute user scenarios Validation The word validation has several related meanings:* In general, validation is the process of checking if something satisfies a certain criterion. Examples would be: checking if a statement is true, if an appliance works as intended, if a computer system is secure, or if computer data is compliant with an open standard. This should not be confused with verification. Verification In the context of hardware and software systems, formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property, using formal methods. Volume Testing Testing which confirms that any values that may become large over time (such as accumulated counts, logs, and data files), can be accommodated by the program and will not cause the program to stop working or degrade its operation in any manner White Box testing / Glass box Testing White box testing, glass box testing or structural testing is used to check that the outputs of a program, given certain inputs, conform to the structural specification of the program. It uses information about the structure of the program to check that it performs correctly. Workflow Testing Scripted end-to-end testing which duplicates specific workflows which are expected to be utilized by the end-user