In the Test Execution app, you can execute the test cases that have been prepared in the Test Preparation app, and track the progress of the test execution.
As a tester, you can execute manual and automated test cases:
Manual Test Cases: Assign statuses and enter comments for individual test actions based on observed application behavior in relation to the expected result, assign defects
Automated Test Cases: Trigger the execution of automated test cases in the connected test automation tool, assign defects
As a testing expert, you can monitor the progress and the outcomes both of manual test runs executed in SAP Cloud ALM, and of automated test runs executed in the connected test automation tool. In detail, the following options are available:
Display a history of all executed manual and automated test runs
Gain insight into all actions that are performed during manual test runs, and into their statuses
View the overall status of a test run, as well as the statuses of individual manual test actions
Access information about detected errors
Navigate to the connected test automation tool for more information
In the test execution, you can execute manual test cases that you've created in the test preparation.
Once the preparation status of your manual test case is Prepared, the test case appears in the Test Execution Overview.
You can then execute it as follows:
This creates a new test run for the selected manual test case.
Carry out the instructions of the individual test actions and check whether the behavior of the application matches the expected result.
Some actions may require you to access the application, which you can do by choosing Application or by selecting an option in the dropdown list.
For each action, assign a status based on the observed result.
You can choose between two Pass statuses and four Fail statuses. If you choose one of the Fail statuses, you are required to enter a comment to specify the error you've observed.
The statuses of the actions influence the overall status of the test case.
If you've found an error, you can create a defect for it in the Defects app and assign it to the affected test runs.
After you've executed all test actions and want to complete the current test run, choose Finish Test Run.
Once you've executed the manual test run, the testing expert can view the results and address potential defects you've discovered.
In the test execution, you can execute automated test cases from the connected test automation tool.
Before an automated test case can appear in the Test Execution Overview, the following prerequisites have to be met:
The connection between SAP Cloud ALM and the test automation tool has been set up successfully, as described in Integrating the Test Automation Tool for SAP S/4HANA Cloud.
The preparation status of the test case has been changed to Prepared.
Once your automated test case appears in the list, you can execute it as follows:
The test execution is now requested in the test automation tool.
You can only execute an automated test case if no execution of this test case is currently running.
If the request is successful, a new test run will be created for the selected automated test case. It is executed directly in the test automation tool.
Once the execution has started, you can track the status and the progress of the test run from the Test Execution Overview.
You can refresh the status and the progress of the test run by choosing (Refresh Test Progress from the Test Automation Tool).
If the automated test case finished with a fail status, you can create a defect for it in the Defects app and assign it to the affected test runs.
Click on the line of the test case to see a list of all of the associated executed test runs.
From here, you can also navigate directly to the test automation tool to view the execution log for more detailed information about potential errors.
How to manage test execution in SAP Cloud ALM without Test Plans.
As of today, the test plan management functionality is not available in SAP Cloud ALM, therefore we need to find ways to manage and structure the testing without this.
To simulate the test planning we can use tasks today, here is a suggestion on how to do it.
First you can create testing tasks as project tasks (or user stories if you prefer working with user stories):
The naming of the task itself is important, we will also use tags for filtering and classification, but I think that a good naming convention is always useful. So, choose one which makes sense to you. In this example, I will use:
- [Test_Phase, aka test plan name]
- [Test Domain], could be finance, sales, …
Now we need to indicate what are the related test cases for this testing task, there are 2 ways of doing this:
- Put the links to the test cases directly in the description of the task
- Use the reference section and add a new reference for each test case
Personally, I like the references, so I will use this approach, but be aware that the sequence of the references cannot be easily changed: the first one that you enter will be in the end the last one in the list as the new references will be added always on top of the list. So, if you wish for more flexibility use the description of the task instead.
Adding a new reference:
Now we need to capture the URL of the test case to execute, for this my recommendation is to open a different tab in your browser and go to the test execution (quick hint, if you use the click of your mouse wheel on the SAP logo at the top left of the page, it will open the SAP Cloud ALM launchpad in a new tab).
Here is the test execution list:
The best way to capture the URL of a test case is to select it in the list. Selecting the test execution row will cause the execution history panel to open:
And it will cause the URL to change, it will now include the test case GUID, and this is the URL that you should now copy to put it into the testing task:
So, let’s copy the URL and paste it into the reference section:
You can then add as many references as you want/need for your testing.
Let’s finally look into the “Additional Information” section of the testing task. It contains a ton of useful information to structure and manage your testing:
1. The scope
The scope of the testing task should be chosen wisely. Remember that, at this moment, the test cases in SAP Cloud ALM are strongly bound to a project and a scope. This will very likely change in the future, but this is currently the way it is, and we need to work with it.
If you have only one scope, then it is an easy one, just select it. If you have structured your project with different scopes then your test cases might be in different scopes, here my recommendation is to be consistent and create your testing tasks with the same scopes as the test cases. There is no constraint if you don’t want to do it, it is just in case you use filters in SAP Cloud ALM and you might get confused sometimes if there is a filter applied on a scope and you are navigating via a link to another scope you might not see exactly what you expected.
If you have testing tasks which will include test cases which belong to different scopes, then do not assign this task to a scope, leave the field empty.
There is no specific recommendation here, it depends if your organization/project uses the workstreams or not. If you use workstreams, then make sure to also assign workstreams to your testing tasks.
Same approach as the workstreams. It depends if within your project you already use deliverables or not.
We can directly establish relationships between requirements and test cases and I would leverage this functionality. I will highlight in a separate document how to do this.
5. Status and Planning
Pretty much all fields should be leveraged here (except maybe for the story points, although they could be used as well to estimate the expected effort for the testing).
- Status: should be used to indicate the progress of the testing
- Priority: to be used as well
- Release: can be used if your project uses releases and if your testing is to be executed for a certain release
- Timebox: should be used, especially if your project uses sprints
- Start and Due dates: to be used
- Assignment: You should use the one which is the most relevant in your project, the assignee allows for direct assignment, whereas the team and role will make the task visible/actionable by several users
- Assigned Role
This is when you will really create the test specific tags. You can be as creative as you want in defining and assigning the tags to the testing task. You could for instance create tags for regression tests, or tags for a given scenario, or tags to manage the assignment, all freedom is allowed. One recommendation would definitely be to create a tag for the testing tasks (#TESTINGTASK for example), so that you could easily leverage this as a filter in the task list or in the overview page, hence creating with close to zero effort a dedicated reporting on your testing.
You can also create subtasks for a project task (or a user story), to break down the work even further and get a great visibility in your testing progress. By having a project task with subtask, you can manage the assignment of test cases to testers very precisely, we could even imagine that you would create a project task to support the testing of a very complex end to end scenario, using either multiple test cases or a very long one, and then you could create sub-tasks for each of the sub-parts of this end to end process and assign the subtasks to dedicated persons (or teams) across the organization.
In the end, the tasks are really like a Swiss-army knife, they give you great flexibility to structure your testing, execute it in the way that you want and finally get a clear reporting on the testing progress.
Keep in mind that:
- The test cases can be assigned to requirements and user stories for traceability: https://blogs.sap.com/2022/02/24/associating-test-cases-and-requirements-in-sap-cloud-alm/
- Defects can be created to test cases: https://blogs.sap.com/2022/06/15/learn-about-defect-management-in-sap-cloud-alm/
With this we can really take the testing one step further with SAP Cloud ALM.
The next significant milestone will be the delivery of test plan management, which will help you to achieve what we have discussed in this post at a much lower cost, with more reusability as well as better test separation and reporting.
You can get an overview of all available test cases of your project, along with relevant information about the current or the most recent execution of the test case.
Tip: If this list is empty, no test cases are currently available for testing. You can create test cases in the Test Preparation app.
Based on your project role, you can perform the following activities here:
As a tester, you can execute manual and automated test cases.
For manual test cases, simply choose Execute and execute a test run right here, in this app. For more information, have a look at the section Executing Manual Test Cases.
The execution of automated test cases can also be triggered by choosing Execute, but takes place directly in the test automation tool. For more information, have a look at the section Executing Automated Test Cases
As a testing expert, you can follow the progress of the test execution.
To take a look at a history of all performed test runs, select a test case row. From there, you can navigate directly into the test run or, in the case of automated test cases, to the test automation tool.
For manual test cases, you can also gain deeper insights by navigating to the open or last closed test run and looking at the statuses and comments for individual test actions.
You can go back and edit your test cases by choosing (Edit in Test Preparation App).
You can see the data variants for your automated test cases as they've been defined in the connected test automation tool.
Some test automation tools don't use the concept of data variants, in which case this column is empty.
If a data variant has been deleted since the last test run, it is labeled as Obsolete and is no longer accessible or executable. You can still view past executions for this variant under Executed Test Runs.