Test analysis is an essential and crucial activity in the testing process. A well-organized test analysis provides an adequate level of confidence in the overall effectiveness of the testing and contributes to the delivery of high-quality software.
Test Analysis Levels
Test analysis can be conducted at different levels – at the Product (system) level and the Feature level (a specific functionality of the system). The difference lies in the scope and focus of the analysis.
Product Test Analysis:
- The goal is to understand the system’s overall functionalities, their priorities, and dependencies.
- It facilitates the development of test plans and test strategies and provides a foundation for regression testing planning.
Feature Test Analysis:
- The goal is to understand the specific feature’s requirements to plan and design targeted tests that verify its correctness and completeness.
- Here we are using test design techniques and creating deliverables for specific features like test cases, test data, and required test environment configurations.
In this article, we will learn about product test analysis, which is a foundation for further analysis of specific features, and feature test analysis we will learn in the next article.
Test Analysis at the Product Level
There are three main activities of test analysis at the Product level:
- Decomposition – breaking down the system into high-level modules.
- Prioritization – determining the importance of the system’s features based on user needs and business goals.
- Dependency Analysis – identifying relations between features in the system.
Stage 1: Decomposition
The first activity is decomposition, which means breaking down the system into smaller parts. Besides that it helps manage the complexity of software development, it also benefits the testing process in the following ways:
- Increasing our understanding of the system’s behavior.
- Facilitating test planning, estimating, and tracking the work progress.
- Giving the capability to evaluate test coverage.
- Providing the ability to perform dependency analysis.
Decomposition can be made at different levels depending on the context and the specific needs of the project. There are decomposition levels like epics, use cases, web pages/screens, modules, objects, etc. These decomposition levels are not mutually exclusive, and they can complement each other. For example, a system may be decomposed into several use cases, and each use case may involve multiple web pages.
A common approach is a decomposition at the object level. Objects represent the entities or concepts in the system that have distinct parameters (properties) and actions (behaviors).
For test analysis at the Product level, we will analyze objects and their actions.
Let’s look at the steps of the product decomposition using the Slack application as an example.
Step 1: Define objects
Here are some objects that we can define:
Can all functionalities in the system be broken down into objects? No, and as was mentioned above, we can combine different decomposition levels. We will use Epics in addition to Objects to describe other functionalities. An epic represents a substantial feature or a collection of related features.
Step 2: Define actions for each object
The next step is defining actions that can be carried out on the object or by the object itself.
Step 3: Define alternative methods for actions
There can be different ways to perform the same action in the system, which should be noted.
Stage 2: Prioritization
Not all features are equally valuable and important for users. Having a list of prioritized functionalities allows us to:
- Start testing more important features, so if there is not enough time to test all we can leave less important features out of the testing scope.
- Find more critical bugs earlier and have enough time to fix them and prevent delays in delivering.
- Create different test suites (for example, a smoke testing suite with critical features, and a regression testing suite with critical and major features).
Step 4: Define priorities for each object/epic
A more advanced approach is to prioritize each action of an object as well.
Stage 3: Dependency Analysis
Most of the features in the system often have relationships with other features. And, the next important step is dependency analysis, which involves identifying the relationships between defined objects/epics, understanding how they interact with each other.
Dependency analysis helps to:
- Identify related features in the system and ensure that all necessary scenarios are covered during testing.
- Identify the potential impact of changes or updates to a specific feature on other related features.
- Assess the potential impact of existing defects on other dependent features and perform targeted testing to mitigate such risks.
Dependency analysis is often documented as a matrix. The matrix consists of rows and columns, where each row and column represents an object (or other item that was used for decomposition). The cells in the matrix indicate the dependency between them.
So, what we need to do, is review our list of objects/epics and think about how they interact with each other, looking for any dependencies such as:
- Functional dependencies - one object relies on the output or behavior of another.
- Data dependencies - one object requires specific data from another one.
Step 5: Define dependencies between objects/epics
Here is an example of the dependency analysis matrix.
This is a one-directional matrix that uses only half of the matrix, where green cells highlight the relationships between items. Additional columns or notes can be used to document the specifics of each dependency.
A more advanced approach is a two-directional matrix that analyzes both sides of the relationships, showing which items depend on others and which items cause others to depend.
The items at the top in the columns depend on the items at the left in the rows, and accordingly, the items in the rows affect the items in the columns. For example, we can see that the Messages object depends on Users, Workspaces, and Channels, and affects History and Search.
This type of matrix provides a more comprehensive view with detailed analysis. The choice between one-directional and two-directional matrices depends on the level of detail required for the project and recommended for safety-critical software.
Let’s review the algorithm of test analysis at the Product level:
All these activities can be performed in the following subsequence:
1. Gather information based on:
- Product documentation (requirements, user stories, user manuals, etc.).
- Product itself (exploratory testing).
- Communication with analysts, developers, stakeholders, etc.
2. Visualize the information
- Different types can be used, such as tables, mind maps, diagrams, etc.
3. Review
- Review the outputs with the team and stakeholders to find missed or incorrect information and make appropriate corrections.
4. Keep it up to date
- As the product evolves, continue to update and maintain the documentation, otherwise it isn’t worth the effort and will get outdated and useless quickly.