Keywords
acceptance criteria, acceptance test-driven development, black-box test technique, boundary value analysis, branch coverage, checklist-based testing, collaboration-based test approach, coverage, coverage item, decision table testing, equivalence partitioning, error guessing, experience-based test technique, exploratory testing, state transition testing, statement coverage, test technique, white-box test technique
Learning Objectives for Chapter 4:
4.1 Test Techniques Overview
- FL-4.1.1 (K2) Distinguish black-box, white-box and experience-based test techniques
4.2 Black-box Test Techniques
- FL-4.2.1 (K3) Use equivalence partitioning to derive test cases
- FL-4.2.2 (K3) Use boundary value analysis to derive test cases
- FL-4.2.3 (K3) Use decision table testing to derive test cases
- FL-4.2.4 (K3) Use state transition testing to derive test cases
4.3 White-box Test Techniques
- FL-4.3.1 (K2) Explain statement testing
- FL-4.3.2 (K2) Explain branch testing
- FL-4.3.3 (K2) Explain the value of white-box testing
4.4 Experience-based Test Techniques
- FL-4.4.1 (K2) Explain error guessing
- FL-4.4.2 (K2) Explain exploratory testing
- FL-4.4.3 (K2) Explain checklist-based testing
4.5 Collaboration-based Test Approaches
- FL-4.5.1 (K2) Explain how to write user stories in collaboration with developers and business representatives
- FL-4.5.2 (K2) Classify the different options for writing acceptance criteria
- FL-4.5.3 (K3) Use acceptance test-driven development (ATDD) to derive test cases
4.1 Test Techniques Overview
Test techniques support the tester in test analysis (what to test) and in test design (how to test). Test techniques help to develop a relatively small, but sufficient, set of test cases in a systematic way. Test techniques also help the tester to define test conditions, identify coverage items, and identify test data during the test analysis and design. Further information on test techniques and their corresponding measures can be found in the ISO/IEC/IEEE 29119-4 standard, and in (Beizer 1990, Craig 2002, Copeland 2004, Koomen 2006, Jorgensen 2014, Ammann 2016, Forgács 2019).
In this syllabus, test techniques are classified as black-box, white-box, and experience-based.
Black-box Test Techniques
Black-box test techniques (also known as specification-based techniques) are based on an analysis of the specified behavior of the test object without reference to its internal structure. Therefore, the test cases are independent of how the software is implemented. Consequently, if the implementation changes, but the required behavior stays the same, then the test cases are still useful.
White-box Test Techniques
White-box test techniques (also known as structure-based techniques) are based on an analysis of the test object’s internal structure and processing. As the test cases are dependent on how the software is designed, they can only be created after the design or implementation of the test object.
Experience-based Test Techniques
Experience-based test techniques effectively use the knowledge and experience of testers for the design and implementation of test cases. The effectiveness of these techniques depends heavily on the tester’s skills. Experience-based test techniques can detect defects that may be missed using the black-box and white-box test techniques. Hence, experience-based test techniques are complementary to the black-box and white-box test techniques.
4.2 Black-Box Test Techniques
Commonly used black-box test techniques discussed in the following sections are:
- Equivalence Partitioning
- Boundary Value Analysis
- Decision Table Testing
- State Transition Testing
4.2.1 Equivalence Partitioning
Equivalence Partitioning (EP) divides data into partitions (known as equivalence partitions) based on the expectation that all the elements of a given partition are to be processed in the same way by the test object. The theory behind this technique is that if a test case, that tests one value from an equivalence partition, detects a defect, this defect should also be detected by test cases that test any other value from the same partition. Therefore, one test for each partition is sufficient.
Equivalence partitions can be identified for any data element related to the test object, including inputs, outputs, configuration items, internal values, time-related values, and interface parameters. The partitions may be continuous or discrete, ordered or unordered, finite or infinite. The partitions must not overlap and must be non-empty sets.
For simple test objects EP can be easy, but in practice, understanding how the test object will treat different values is often complicated. Therefore, partitioning should be done with care.
A partition containing valid values is called a valid partition. A partition containing invalid values is called an invalid partition. The definitions of valid and invalid values may vary among teams and organizations. For example, valid values may be interpreted as those that should be processed by the test object or as those for which the specification defines their processing. Invalid values may be interpreted as those that should be ignored or rejected by the test object or as those for which no processing is defined in the test object specification.
In EP, the coverage items are the equivalence partitions. To achieve 100% coverage with this technique, test cases must exercise all identified partitions (including invalid partitions) by covering each partition at least once. Coverage is measured as the number of partitions exercised by at least one test case, divided by the total number of identified partitions, and is expressed as a percentage.
Many test objects include multiple sets of partitions (e.g., test objects with more than one input parameter), which means that a test case will cover partitions from different sets of partitions. The simplest coverage criterion in the case of multiple sets of partitions is called Each Choice coverage (Ammann 2016). Each Choice coverage requires test cases to exercise each partition from each set of partitions at least once. Each Choice coverage does not take into account combinations of partitions.
4.2.2. Boundary Value Analysis
Boundary Value Analysis (BVA) is a technique based on exercising the boundaries of equivalence partitions. Therefore, BVA can only be used for ordered partitions. The minimum and maximum values of a partition are its boundary values. In the case of BVA, if two elements belong to the same partition, all elements between them must also belong to that partition.
BVA focuses on the boundary values of the partitions because developers are more likely to make errors with these boundary values. Typical defects found by BVA are located where implemented boundaries are misplaced to positions above or below their intended positions or are omitted altogether.
This syllabus covers two versions of the BVA: 2-value and 3-value BVA. They differ in terms of coverage items per boundary that need to be exercised to achieve 100% coverage.
In 2-value BVA (Craig 2002, Myers 2011), for each boundary value there are two coverage items: this boundary value and its closest neighbor belonging to the adjacent partition. To achieve 100% coverage with 2-value BVA, test cases must exercise all coverage items, i.e., all identified boundary values. Coverage is measured as the number of boundary values that were exercised, divided by the total number of identified boundary values, and is expressed as a percentage.
In 3-value BVA (Koomen 2006, O’Regan 2019), for each boundary value there are three coverage items: this boundary value and both its neighbors. Therefore, in 3-value BVA some of the coverage items may not be boundary values. To achieve 100% coverage with 3-value BVA, test cases must exercise all coverage items, i.e., identified boundary values and their neighbors. Coverage is measured as the number of boundary values and their neighbors exercised, divided by the total number of identified boundary values and their neighbors, and is expressed as a percentage.
3-value BVA is more rigorous than 2-value BVA as it may detect defects overlooked by 2-value BVA. For example, if the decision “if (x ≤ 10) …” is incorrectly implemented as “if (x = 10) …”, no test data derived from the 2-value BVA (x = 10, x = 11) can detect the defect. However, x = 9, derived from the 3-value BVA, is likely to detect it.
4.2.3. Decision Table Testing
Decision tables are used for testing the implementation of system requirements that specify how different combinations of conditions result in different outcomes. Decision tables are an effective way of recording complex logic, such as business rules.
When creating decision tables, the conditions and the resulting actions of the system are defined. These form the rows of the table. Each column corresponds to a decision rule that defines a unique combination of conditions, along with the associated actions. In limited-entry decision tables all the values of the conditions and actions (except for irrelevant or infeasible ones; see below) are shown as Boolean values (true or false). Alternatively, in extended-entry decision tables some or all the conditions and actions may also take on multiple values (e.g., ranges of numbers, equivalence partitions, discrete values).
The notation for conditions is as follows: “T” (true) means that the condition is satisfied. “F” (false) means that the condition is not satisfied. “–” means that the value of the condition is irrelevant for the action outcome. “N/A” means that the condition is infeasible for a given rule. For actions: “X” means that the action should occur. Blank means that the action should not occur. Other notations may also be used.
A full decision table has enough columns to cover every combination of conditions. The table can be simplified by deleting columns containing infeasible combinations of conditions. The table can also be minimized by merging columns, in which some conditions do not affect the outcome, into a single column. Decision table minimization algorithms are out of scope of this syllabus.
In decision table testing, the coverage items are the columns containing feasible combinations of conditions. To achieve 100% coverage with this technique, test cases must exercise all these columns. Coverage is measured as the number of exercised columns, divided by the total number of feasible columns, and is expressed as a percentage.
The strength of decision table testing is that it provides a systematic approach to identify all the combinations of conditions, some of which might otherwise be overlooked. It also helps to find any gaps or contradictions in the requirements. If there are many conditions, exercising all the decision rules may be time consuming, since the number of rules grows exponentially with the number of conditions. In such a case, to reduce the number of rules that need to be exercised, a minimized decision table or a riskbased approach may be used.
4.2.4. State Transition Testing
A state transition diagram models the behavior of a system by showing its possible states and valid state transitions. A transition is initiated by an event, which may be additionally qualified by a guard condition. The transitions are assumed to be instantaneous and may sometimes result in the software taking action. The common transition labeling syntax is as follows: “event [guard condition] / action”. Guard conditions and actions can be omitted if they do not exist or are irrelevant for the tester.
A state table is a model equivalent to a state transition diagram. Its rows represent states, and its columns represent events (together with guard conditions if they exist). Table entries (cells) represent transitions, and contain the target state, as well as the resulting actions, if defined. In contrast to the state transition diagram, the state table explicitly shows invalid transitions, which are represented by empty cells.
A test case based on a state transition diagram or state table is usually represented as a sequence of events, which results in a sequence of state changes (and actions, if needed). One test case may, and usually will, cover several transitions between states.
There exist many coverage criteria for state transition testing. This syllabus discusses three of them.
In all states coverage, the coverage items are the states. To achieve 100% all states coverage, test cases must ensure that all the states are visited. Coverage is measured as the number of visited states divided by the total number of states, and is expressed as a percentage.
In valid transitions coverage (also called 0-switch coverage), the coverage items are single valid transitions. To achieve 100% valid transitions coverage, test cases must exercise all the valid transitions. Coverage is measured as the number of exercised valid transitions divided by the total number of valid transitions, and is expressed as a percentage.
In all transitions coverage, the coverage items are all the transitions shown in a state table. To achieve 100% all transitions coverage, test cases must exercise all the valid transitions and attempt to execute invalid transitions. Testing only one invalid transition in a single test case helps to avoid fault masking, i.e., a situation in which one defect prevents the detection of another. Coverage is measured as the number of valid and invalid transitions exercised or attempted to be covered by executed test cases, divided by the total number of valid and invalid transitions, and is expressed as a percentage.
All states coverage is weaker than valid transitions coverage, because it can typically be achieved without exercising all the transitions. Valid transitions coverage is the most widely used coverage criterion. Achieving full valid transitions coverage guarantees full all states coverage. Achieving full all transitions coverage guarantees both full all states coverage and full valid transitions coverage and should be a minimum requirement for mission and safety-critical software.
4.3 White-Box Test Techniques
Because of their popularity and simplicity, this section focuses on two code-related white-box test techniques:
- Statement Testing
- Branch Testing
4.3.1 Statement Testing and Statement Coverage
In statement testing, the coverage items are executable statements. The aim is to design test cases that exercise statements in the code until an acceptable level of coverage is achieved. Coverage is measured as the number of statements exercised by the test cases divided by the total number of executable statements in the code and is expressed as a percentage.
When 100% statement coverage is achieved, it ensures that all executable statements in the code have been exercised at least once. However, exercising a statement with a test case will not detect defects in all cases, such as those that are data-dependent. Additionally, 100% statement coverage does not ensure that all decision logic has been tested, including all branches in the code (see chapter 4.3.2).
4.3.2 Branch Testing and Branch Coverage
A branch is a transfer of control between two nodes in the control flow graph, showing the possible sequences in which source code statements are executed. In branch testing, the coverage items are branches, and the aim is to design test cases to exercise branches in the code until an acceptable level of coverage is achieved. Coverage is measured as the number of branches exercised by the test cases divided by the total number of branches and is expressed as a percentage.
When 100% branch coverage is achieved, all branches in the code, including conditional and unconditional branches, are exercised by test cases. However, exercising a branch with a test case will not detect defects in all cases, such as those requiring the execution of specific paths in the code.
Branch coverage subsumes statement coverage, meaning that any set of test cases achieving 100% branch coverage also achieves 100% statement coverage.
4.3.3 The Value of White-Box Testing
A fundamental strength of all white-box techniques is that the entire software implementation is taken into account during testing, facilitating defect detection even when the software specification is vague, outdated, or incomplete. However, a corresponding weakness is that if the software does not implement one or more requirements, white-box testing may not detect resulting defects of omission.
White-box techniques can be used in static testing, such as during dry runs of code. They are well-suited for reviewing code that is not yet ready for execution, as well as pseudocode and other high-level or top-down logic that can be modeled with a control flow graph.
Performing only black-box testing does not provide a measure of actual code coverage. White-box coverage measures provide an objective measurement of coverage and necessary information to generate additional tests to increase coverage and subsequently increase confidence in the code.
4.4 Experience-based Test Techniques
Commonly used experience-based test techniques discussed in the following sections are:
- Error Guessing
- Exploratory Testing
- Checklist-based Testing
4.4.1 Error Guessing
Error guessing is a technique used to anticipate the occurrence of errors, defects, and failures based on the tester’s knowledge. This includes knowledge of how the application has worked in the past, the types of errors developers tend to make, the types of defects resulting from these errors, and the types of failures that have occurred in other similar applications.
Errors, defects, and failures can be related to various aspects such as input, output, logic, computation, interfaces, or data. Fault attacks are a methodical approach to implementing error guessing, requiring the tester to create or acquire a list of possible errors, defects, and failures and design tests to identify and expose defects associated with the errors or cause the failures.
For more information on error guessing and fault attacks, refer to (Whittaker 2002, Whittaker 2003, Andrews 2006).
4.4.2 Exploratory Testing
In exploratory testing, tests are simultaneously designed, executed, and evaluated while the tester learns about the test object. It is used to gain more knowledge about the test object, explore it deeply with focused tests, and create tests for untested areas.
Exploratory testing can be conducted using a session-based approach, where the testing is conducted within a defined time-box. The tester uses a test charter containing test objectives to guide the testing. After the test session, a debriefing takes place to discuss the test results. Test objectives may be treated as high-level test conditions, and coverage items are identified and exercised during the test session.
Exploratory testing is useful when specifications are limited or inadequate, and there is time pressure on the testing. It can complement other more formal test techniques and is more effective when the tester is experienced, has domain knowledge, and possesses essential skills like analytical skills, curiosity, and creativity.
For more information about exploratory testing, refer to (Kaner 1999, Whittaker 2009, Hendrickson 2013).
4.4.3 Checklist-Based Testing
In checklist-based testing, a tester designs, implements, and executes tests to cover test conditions from a checklist. Checklists can be built based on experience, knowledge about user needs, or an understanding of why and how software fails.
Checklist items are often phrased as questions, and each item can be checked separately. These items may refer to requirements, graphical interface properties, quality characteristics, or other forms of test conditions. Checklists can support various test types, including functional and non-functional testing.
Checklists should be regularly updated based on defect analysis, adding new entries and removing less effective ones over time. However, care should be taken to avoid making the checklist too long.
In the absence of detailed test cases, checklist-based testing can provide guidelines and some degree of consistency for testing. However, it may result in greater coverage but less repeatability due to the variability in the actual testing.
4.5 Collaboration-based Test Approaches
Each of the above-mentioned techniques (see sections 4.2, 4.3, 4.4) has a particular objective with respect to defect detection. Collaboration-based approaches, on the other hand, focus also on defect avoidance through collaboration and communication.
4.5.1 Collaborative User Story Writing
A user story represents a feature that will be valuable to either a user or purchaser of a system or software. User stories have three critical aspects, called the “3 C’s”: Card (the medium describing a user story), Conversation (explains how the software will be used), and Confirmation (the acceptance criteria).
The most common format for a user story is “As a [role], I want [goal to be accomplished], so that I can [resulting business value for the role],” followed by the acceptance criteria.
Collaborative authorship of the user story can use techniques such as brainstorming and mind mapping. The collaboration allows the team to obtain a shared vision of what should be delivered, considering perspectives from business, development, and testing.
Good user stories should be independent, negotiable, valuable, estimable, small, and testable (INVEST). If a stakeholder doesn’t know how to test a user story, it may indicate a lack of clarity, value, or the need for testing assistance.
4.5.2 Acceptance Criteria
Acceptance criteria for a user story are the conditions that an implementation must meet to be accepted by stakeholders. Acceptance criteria define the scope of the user story, reach consensus among stakeholders, describe positive and negative scenarios, serve as a basis for acceptance testing, and allow accurate planning and estimation.
Acceptance criteria can be written in scenario-oriented format (e.g., Given/When/Then format used in BDD) or rule-oriented format (e.g., bullet point verification list or tabulated form of input-output mapping). Most acceptance criteria can be documented in one of these two formats, but custom formats are acceptable as long as the criteria are well-defined and unambiguous.
4.5.3 Acceptance Test-driven Development (ATDD)
ATDD is a test-first approach where test cases are created prior to implementing the user story. The test cases are created by team members with different perspectives such as customers, developers, and testers. Test cases can be executed manually or automated.
The process starts with a specification workshop to analyze, discuss, and write the user story and its acceptance criteria. Incompleteness, ambiguities, or defects in the user story are resolved during this workshop. The next step is creating the test cases, which can be done by the entire team or an individual tester. The test cases are based on the acceptance criteria and serve as examples of how the software works, helping the team implement the user story correctly.
During test design, the test techniques described in sections 4.2, 4.3, and 4.4 may be applied. Test cases should cover positive scenarios first, confirming correct behavior without exceptions or error conditions. Negative testing should be performed after positive test cases, and non-functional quality characteristics (e.g., performance efficiency, usability) should also be covered.
Test cases should be expressed in a way that stakeholders can understand, typically using natural language sentences involving preconditions, inputs, and postconditions. The test cases must cover all the characteristics of the user story without going beyond its scope. Additionally, no two test cases should describe the same user story characteristics.
When captured in a format supported by a test automation framework, developers can automate the test cases by writing supporting code while implementing the feature described by a user story. The acceptance tests then become executable requirements.