Keywords
Defect management, defect report, entry criteria, exit criteria, product risk, project risk, risk, risk analysis, risk assessment, risk control, risk identification, risk level, risk management, risk mitigation, risk monitoring, risk-based testing, test approach, test completion report, test control, test monitoring, test plan, test planning, test progress report, test pyramid, testing quadrants
Learning Objectives for Chapter 5:
5.1 Test Planning
- (K2) Exemplify the purpose and content of a test plan
- (K1) Recognize how a tester adds value to iteration and release planning
- (K2) Compare and contrast entry criteria and exit criteria
- (K3) Use estimation techniques to calculate the required test effort
- (K3) Apply test case prioritization
- (K1) Recall the concepts of the test pyramid
- (K2) Summarize the testing quadrants and their relationships with test levels and test types
5.2 Risk Management
- (K1) Identify risk level by using risk likelihood and risk impact
- (K2) Distinguish between project risks and product risks
- (K2) Explain how product risk analysis may influence thoroughness and scope of testing
- (K2) Explain what measures can be taken in response to analyzed product risks
5.3 Test Monitoring, Test Control and Test Completion
- (K1) Recall metrics used for testing
- (K2) Summarize the purposes, content, and audiences for test reports
- (K2) Exemplify how to communicate the status of testing
5.4 Configuration Management
- (K2) Summarize how configuration management supports testing
5.5 Defect Management
- (K3) Prepare a defect report
5.1 Test Planning
5.1.1 Purpose and Content of a Test Plan
A test plan describes the objectives, resources, and processes for a test project. A test plan:
- Documents the means and schedule for achieving test objectives
- Helps to ensure that the performed test activities will meet the established criteria
- Serves as a means of communication with team members and other stakeholders
- Demonstrates that testing will adhere to the existing test policy and test strategy (or explains why the testing will deviate from them)
Test planning guides the testers’ thinking and forces the testers to confront the future challenges related to risks, schedules, people, tools, costs, effort, etc. The process of preparing a test plan is a useful way to think through the efforts needed to achieve the test project objectives.
The typical content of a test plan includes:
- Context of testing (e.g., scope, test objectives, constraints, test basis)
- Assumptions and constraints of the test project
- Stakeholders (e.g., roles, responsibilities, relevance to testing, hiring and training needs)
- Communication (e.g., forms and frequency of communication, documentation templates)
- Risk register (e.g., product risks, project risks)
- Test approach (e.g., test levels, test types, test techniques, test deliverables, entry criteria and exit criteria, independence of testing, metrics to be collected, test data requirements, test environment requirements, deviations from the organizational test policy and test strategy)
- Budget and schedule
More details about the test plan and its content can be found in the ISO/IEC/IEEE 29119-3 standard.
5.1.2 Tester’s Contribution to Iteration and Release Planning
In iterative SDLCs, typically two kinds of planning occur: release planning and iteration planning.
Release planning looks ahead to the release of a product, defines and re-defines the product backlog, and may involve refining larger user stories into a set of smaller user stories. It also serves as the basis for the test approach and test plan across all iterations. Testers involved in release planning participate in writing testable user stories and acceptance criteria (see section 4.5), participate in project and quality risk analyses (see section 5.2), estimate test effort associated with user stories (see section 5.1.4), determine the test approach, and plan the testing for the release.
Iteration planning looks ahead to the end of a single iteration and is concerned with the iteration backlog. Testers involved in iteration planning participate in the detailed risk analysis of user stories, determine the testability of user stories, break down user stories into tasks (particularly testing tasks), estimate test effort for all testing tasks, and identify and refine functional and non-functional aspects of the test object.
5.1.3 Entry Criteria and Exit Criteria
Entry criteria define the preconditions for undertaking a given activity. If entry criteria are not met, it is likely that the activity will prove to be more difficult, time-consuming, costly, and riskier. Exit criteria define what must be achieved in order to declare an activity completed. Entry criteria and exit criteria should be defined for each test level and will differ based on the test objectives.
Typical entry criteria include: availability of resources (e.g., people, tools, environments, test data, budget, time), availability of testware (e.g., test basis, testable requirements, user stories, test cases), and initial quality level of a test object (e.g., all smoke tests have passed).
Typical exit criteria include: measures of thoroughness (e.g., achieved level of coverage, number of unresolved defects, defect density, number of failed test cases), and completion criteria (e.g., planned tests have been executed, static testing has been performed, all defects found are reported, all regression tests are automated).
Running out of time or budget can also be viewed as valid exit criteria. Even without other exit criteria being satisfied, it can be acceptable to end testing under such circumstances if the stakeholders have reviewed and accepted the risk to go live without further testing.
In Agile software development, exit criteria are often called Definition of Done, defining the team’s objective metrics for a releasable item. Entry criteria that a user story must fulfill to start the development and/or testing activities are called Definition of Ready.
5.1.4 Estimation Techniques
Test effort estimation involves predicting the amount of test-related work needed to meet the objectives of a test project. It is important to make it clear to the stakeholders that the estimate is based on a number of assumptions and is always subject to estimation error. Estimation for small tasks is usually more accurate than for the large ones. Therefore, when estimating a large task, it can be decomposed into a set of smaller tasks which then in turn can be estimated.
In this syllabus, the following four estimation techniques are described:
- Estimation based on ratios: In this metrics-based technique, figures are collected from previous projects within the organization, which makes it possible to derive “standard” ratios for similar projects.
- Extrapolation: In this metrics-based technique, measurements are made as early as possible in the current project to gather the data. Having enough observations, the effort required for the remaining work can be approximated by extrapolating this data (usually by applying a mathematical model).
- Wideband Delphi: In this iterative, expert-based technique, experts make experience-based estimations. Each expert, in isolation, estimates the effort. The results are collected, and if there are deviations that are out of the range of the agreed-upon boundaries, the experts discuss their current estimates. Each expert is then asked to make a new estimation based on that feedback, again in isolation. This process is repeated until a consensus is reached.
- Three-point estimation: In this expert-based technique, three estimations are made by the experts: the most optimistic estimation (a), the most likely estimation (m), and the most pessimistic estimation (b). The final estimate (E) is their weighted arithmetic mean.
See (Kan 2003, Koomen 2006, Westfall 2009) for these and many other test estimation techniques.
5.1.5 Test Case Prioritization
Once the test cases and test procedures are specified and assembled into test suites, these test suites can be arranged in a test execution schedule that defines the order in which they are to be run. When prioritizing test cases, different factors can be taken into account. The most commonly used test case prioritization strategies are as follows:
- Risk-based prioritization: The order of test execution is based on the results of risk analysis (see section 5.2.3). Test cases covering the most important risks are executed first.
- Coverage-based prioritization: The order of test execution is based on coverage (e.g., statement coverage). Test cases achieving the highest coverage are executed first. In another variant, called additional coverage prioritization, the test case achieving the highest coverage is executed first; each subsequent test case is the one that achieves the highest additional coverage.
- Requirements-based prioritization: The order of test execution is based on the priorities of the requirements traced back to the corresponding test cases. Requirement priorities are defined by stakeholders. Test cases related to the most important requirements are executed first.
Ideally, test cases would be ordered to run based on their priority levels, using one of the above-mentioned prioritization strategies. However, this practice may not work if the test cases or the features being tested have dependencies. If a test case with a higher priority is dependent on a test case with a lower priority, the lower priority test case must be executed first.
The order of test execution must also take into account the availability of resources. For example, the required test tools, test environments, or people may only be available for a specific time window.
5.1.6 Test Pyramid
The test pyramid is a model showing that different tests may have different granularity. The test pyramid model supports the team in test automation and in test effort allocation by showing that different goals are supported by different levels of test automation. The pyramid layers represent groups of tests. The higher the layer, the lower the test granularity, test isolation, and test execution time. Tests in the bottom layer are small, isolated, fast, and check a small piece of functionality, so usually a lot of them are needed to achieve reasonable coverage. The top layer represents complex, high-level, end-to-end tests. These high-level tests are generally slower than the tests from the lower layers, and they typically check a large piece of functionality, so usually just a few of them are needed to achieve reasonable coverage. The number and naming of the layers may differ. For example, the original test pyramid model (Cohn 2009) defines three layers: “unit tests,” “service tests,” and “UI tests.” Another popular model defines unit (component) tests, integration (component integration) tests, and end-to-end tests. Other test levels (see section 2.2.1) can also be used.
5.1.7 Testing Quadrants
The testing quadrants, defined by Brian Marick (Marick 2003, Crispin 2008), group the test levels with the appropriate test types, activities, test techniques, and work products in Agile software development. The model supports test management in visualizing these to ensure that all appropriate test types and test levels are included in the SDLC and in understanding that some test types are more relevant to certain test levels than others. This model also provides a way to differentiate and describe the types of tests to all stakeholders, including developers, testers, and business representatives.
In this model, tests can be business facing or technology facing. Tests can also support the team (i.e., guide the development) or critique the product (i.e., measure its behavior against the expectations). The combination of these two viewpoints determines the four quadrants:
- Quadrant Q1 (technology facing, support the team): This quadrant contains component and component integration tests. These tests should be automated and included in the CI process.
- Quadrant Q2 (business facing, support the team): This quadrant contains functional tests, examples, user story tests, user experience prototypes, API testing, and simulations. These tests check the acceptance criteria and can be manual or automated.
- Quadrant Q3 (business facing, critique the product): This quadrant contains exploratory testing, usability testing, user acceptance testing. These tests are user-oriented and often manual.
- Quadrant Q4 (technology facing, critique the product): This quadrant contains smoke tests and non-functional tests (except usability tests). These tests are often automated.
5.2 Risk Management
Organizations face many internal and external factors that make it uncertain whether and when they will achieve their objectives (ISO 31000). Risk management allows the organizations to increase the likelihood of achieving objectives, improve the quality of their products, and increase stakeholders’ confidence and trust.
The main risk management activities are:
- Risk analysis (consisting of risk identification and risk assessment; see section 5.2.3)
- Risk control (consisting of risk mitigation and risk monitoring; see section 5.2.4)
The test approach, in which test activities are selected, prioritized, and managed based on risk analysis and risk control, is called risk-based testing.
5.2.1 Risk Definition and Risk Attributes
Risk is a potential event, hazard, threat, or situation whose occurrence causes an adverse effect. A risk can be characterized by two factors:
- Risk likelihood: the probability of the risk occurrence (greater than zero and less than one)
- Risk impact (harm): the consequences of this occurrence
These two factors express the risk level, which is a measure for the risk. The higher the risk level, the more important is its treatment.
5.2.2 Project Risks and Product Risks
In software testing, one is generally concerned with two types of risks: project risks and product risks.
Project risks are related to the management and control of the project. Project risks include:
- Organizational issues (e.g., delays in work product deliveries, inaccurate estimates, cost-cutting)
- People issues (e.g., insufficient skills, conflicts, communication problems, shortage of staff)
- Technical issues (e.g., scope creep, poor tool support)
- Supplier issues (e.g., third-party delivery failure, bankruptcy of the supporting company)
Project risks, when they occur, may have an impact on the project schedule, budget, or scope, which affects the project’s ability to achieve its objectives.
Product risks are related to the product quality characteristics (e.g., described in the ISO 25010 quality model). Examples of product risks include missing or wrong functionality, incorrect calculations, runtime errors, poor architecture, inefficient algorithms, inadequate response time, poor user experience, security vulnerabilities. Product risks, when they occur, may result in various negative consequences, including user dissatisfaction, loss of revenue, trust, reputation, damage to third parties, high maintenance costs, overload of the helpdesk, criminal penalties, and in extreme cases, physical damage, injuries, or even death.
5.2.3 Product Risk Analysis
From a testing perspective, the goal of product risk analysis is to provide awareness of product risk in order to focus the testing effort in a way that minimizes the residual level of product risk. Ideally, product risk analysis begins early in the SDLC.
Product risk analysis consists of risk identification and risk assessment. Risk identification is about generating a comprehensive list of risks. Stakeholders can identify risks by using various techniques and tools, e.g., brainstorming, workshops, interviews, or cause-effect diagrams. Risk assessment involves the categorization of identified risks, determining their risk likelihood, risk impact and level, prioritizing, and proposing ways to handle them. Categorization helps in assigning mitigation actions because usually risks falling into the same category can be mitigated using a similar approach.
Risk assessment can use a quantitative or qualitative approach or a mix of them. In the quantitative approach, the risk level is calculated as the multiplication of risk likelihood and risk impact. In the qualitative approach, the risk level can be determined using a risk matrix.
Product risk analysis may influence the thoroughness and scope of testing. Its results are used to:
- Determine the scope of testing to be carried out
- Determine the particular test levels and propose test types to be performed
- Determine the test techniques to be employed and the coverage to be achieved
- Estimate the test effort required for each task
- Prioritize testing in an attempt to find the critical defects as early as possible
- Determine whether any activities in addition to testing could be employed to reduce risk
5.2.4 Product Risk Control
Product risk control comprises all measures that are taken in response to identified and assessed product risks. Product risk control consists of risk mitigation and risk monitoring. Risk mitigation involves implementing the actions proposed in risk assessment to reduce the risk level. The aim of risk monitoring is to ensure that the mitigation actions are effective, to obtain further information to improve risk assessment, and to identify emerging risks.
With respect to product risk control, once a risk has been analyzed, several response options to risk are possible, e.g., risk mitigation by testing, risk acceptance, risk transfer, or contingency plan (Veenendaal 2012). Actions that can be taken to mitigate the product risks by testing are as follows:
- Select the testers with the right level of experience and skills, suitable for a given risk type
- Apply an appropriate level of independence of testing
- Conduct reviews and perform static analysis
- Apply the appropriate test techniques and coverage levels
- Apply the appropriate test types addressing the affected quality characteristics
- Perform dynamic testing, including regression testing
5.3 Test Monitoring, Test Control and Test Completion
Test monitoring is the process of gathering information about testing to assess test progress and measure whether the test exit criteria or associated tasks have been satisfied. It involves tracking the testing activities, evaluating the coverage of product risks, requirements, and acceptance criteria. The information collected during test monitoring is used in test control to provide guidance, corrective actions, and control directives for effective and efficient testing.
Examples of control directives in test control include:
- Reprioritizing tests when identified risks become issues
- Re-evaluating whether a test item meets entry or exit criteria due to rework
- Adjusting the test schedule to address delays in test environment delivery
- Adding new resources as needed
Test completion involves collecting data from completed test activities to consolidate experience, testware, and relevant information. Test completion activities occur at project milestones, such as the completion of a test level, an agile iteration, a test project (or its cancellation), the release of a software system, or the completion of a maintenance release.
5.3.1 Metrics used in Testing
Test metrics are gathered to show progress against the planned schedule and budget, assess the current quality of the test object, and evaluate the effectiveness of the test activities. Test monitoring collects various metrics to support test control and test completion. Some common test metrics include:
- Project progress metrics (e.g., task completion, resource usage, test effort)
- Test progress metrics (e.g., test case implementation progress, test environment preparation progress, number of test cases run/not run, passed/failed, test execution time)
- Product quality metrics (e.g., availability, response time, mean time to failure)
- Defect metrics (e.g., number and priorities of defects found/fixed, defect density, defect detection percentage)
- Risk metrics (e.g., residual risk level)
- Coverage metrics (e.g., requirements coverage, code coverage)
- Cost metrics (e.g., cost of testing, organizational cost of quality)
5.3.2 Purpose, Content, and Audience for Test Reports
Test reporting is the process of summarizing and communicating test information during and after testing. Test progress reports support ongoing control of testing and provide enough information to make modifications to the test schedule, resources, or test plan based on deviations from the plan or changed circumstances. Test completion reports summarize a specific stage of testing and provide information for subsequent testing.
Test progress reports include:
- Test period
- Test progress (ahead or behind schedule) and notable deviations
- Impediments for testing and their workarounds
- Test metrics (see section 5.3.1 for examples)
- New and changed risks within the testing period
- Testing planned for the next period
Test completion reports are prepared when a project, test level, or test type is complete, and its exit criteria have been met. These reports utilize test progress reports and other data. Typical test completion reports include:
- Test summary
- Testing and product quality evaluation based on the original test plan
- Deviations from the test plan (schedule, duration, effort)
- Testing impediments and workarounds
- Test metrics based on test progress reports
- Unmitigated risks and unresolved defects
- Lessons learned relevant to the testing
Different audiences require different information in the reports, which influences the degree of formality and the frequency of reporting. Reporting on test progress to team members is often frequent and informal, while reporting on completed projects follows a set template and occurs only once. The ISO/IEC/IEEE 29119-3 standard provides templates and examples for test progress reports (test status reports) and test completion reports.
5.3.3 Communicating the Status of Testing
The means of communicating test status vary depending on test management concerns, organizational test strategies, regulatory standards, and the nature of the team. Options for communicating test status include:
- Verbal communication with team members and stakeholders
- Dashboards (e.g., CI/CD dashboards, task boards, burn-down charts)
- Electronic communication channels (e.g., email, chat)
- Online documentation
- Formal test reports (see section 5.3.2)
One or more of these options can be used. More formal communication may be necessary for distributed teams where face-to-face communication is limited. Tailoring the communication to different stakeholders’ interests is important as they may require different types of information.
5.4 Configuration Management
In testing, configuration management (CM) is a discipline that involves identifying, controlling, and tracking work products as configuration items. These work products include test plans, test strategies, test conditions, test cases, test scripts, test results, test logs, and test reports.
For complex configuration items, such as test environments, CM records the items it consists of, their relationships, and versions. Once a configuration item is approved for testing, it becomes a baseline and can only be changed through a formal change control process.
Configuration management also keeps a record of changed configuration items when a new baseline is created. This allows the ability to revert to a previous baseline to reproduce previous test results.
To support testing effectively, CM ensures the following:
- All configuration items, including test items, are uniquely identified, version controlled, tracked for changes, and related to other configuration items for traceability throughout the test process.
- All identified documentation and software items are unambiguously referenced in test documentation.
Continuous integration, continuous delivery, continuous deployment, and associated testing are typically implemented as part of an automated DevOps pipeline. Automated CM is normally included in this pipeline.
5.5 Defect Management
Defect management is crucial since one of the primary test objectives is to find defects. An established defect management process is essential to handle reported anomalies. Anomalies may be reported during any phase of the software development life cycle (SDLC) and may turn out to be real defects or something else (e.g., false positive, change request). The defect management process includes a workflow for handling individual anomalies, from their discovery to closure, and rules for their classification.
Typical defect reports have the following objectives:
- Provide sufficient information to those responsible for handling and resolving reported defects
- Track the quality of the work product
- Provide ideas for process improvement
A defect report logged during dynamic testing typically includes:
- Unique identifier
- Title with a short summary of the anomaly
- Date of observation, issuing organization, and author information
- Identification of the test object and test environment
- Context of the defect, including relevant information such as the test technique, checklist, or test data used
- Description of the failure, including steps to reproduce and relevant supporting materials
- Expected results and actual results
- Severity and priority of the defect
- Status of the defect
- References to the test case
Some of this data may be automatically included when using defect management tools. The ISO/IEC/IEEE 29119-3 standard provides document templates for defect reports (referred to as incident reports) and example defect reports.